Best practices for directory structure for SAS deployments

[sethuraman_ram]

Hi, 

This is probably a long question. So please bear with me. 

I would like some guidance from members on how to effectively set up a directory structure for our SAS deployment. To set the context, we currently run SAS on a Linux environment and have about 10 developers working on different client tools such as Enterprise Guide(EGuide) and Enterprise Miner(EMiner) or Visual Analytics(VA). The current issues that we are coming up against are that these developers create content suich as SAS data sets, EGuide reports/projects, EMiner Models and VA queries and I need guidance on:

1. How do I set up a directory structure so that it is intuitive enough for developers to place their content into the respective folders and i can provide the appropriate permissions to those folders
2. The directory structure should also allow for easy migration of content from a DEV workspace to a QA workspace and then on to a PROD workspace. These workspaces will be be on the same Linux environment as we have do not have separate SAS environments for test and prod.
3. What tools I can use for version control so that I can have multiple developers work on the same artefacts independently but can still keep track of the changes made.

I would love to hear from members about their thoughts on this. 

[SASKiwi]

It's a long answer too and beyond the scope of answering completely in a single post!

Sorting out your SAS environments and directory structures should have been done as part of your SAS design prior to installation. I'm assuming that wasn't done in your case? I am also assuming you only have one physical SAS Compute server.

I can only discuss general principles as many of the requirements will be site-specific.

Your DEV, QA and PROD environments should each have their own separate file system and separate disk quotas so that one cannot impact the other. Security on DEV will be most open. Access to QA and PROD will be limited to those who need it. Where I work our PROD area is limited to read-only for most SAS users and write access only to SAS administrators. Folder structures should be based on applications - I'm not a big fan of personal folders.

Ideally you should have a separate SAS App server for each area, called say Appdev, Appqa, Appprod (naming is up to you). This means you are separating the server's resources so a problem with the DEV App server wont affect say the PROD App server. Each of these appservers will have its own SAS WORK disk space again improving reliability. It also means you can have separate metadata setups for each of the areas.

Moving of code between environments is best handled using software tools. SAS Management Console has the ability to move programs between environments. Where I work we prefer to use third-party change management tools as this has a secure respository for all our code and handles all versioning and team collaboration of the software, as well as automatically deploying our PROD code. We use Microsoft's TFS (Team Foundation Server) and Visual Studio and I rate it highly. It integrates well with SAS EG. You should look at what tools your organisation uses though as you will get a lot better support that way.

I'd also recommend that data should generally be only copied PROD ==> QA ==> DEV and NOT the other way. It is important that PROD data should only be processed/sourced by PROD code to maintain its integrity.

That's probably enough for now, but happy to answer more specific questions you may have. 

[LinusH]

Just wish to add to @SASKiwi good advice, that you should not only separate the App-servers, but the whole configurations.

This means separate top directories (Lev1 - Lev2 - Lev3), withe separate metadata servers etc. This means that can separate metadata and other configurations between environments as well , not just the data.

[sethuraman_ram]

Thank you @SASKiwi and @LinusH for your suggestions.We are looking to implement the suggestion of separating respurces for the SasApp servers to model a Dev/QA environment and a Prod environment. Unfortunately,licensing issues restrict us from separating out all the 3 environments.

As far as the directory structures as concerned, we are thinking of adopting the following structure for DEV:

Root/<Business Unit/Function>/Dev/<developer name>/lib

Root/<Business Unit/Function>/Dev/<developer name>/lib/datasets

Root/<Business Unit/Function>/Dev/<developer name>/lib/doc

Root/<Business Unit/Function>/Dev/<developer name>/lib/Infiles

Root/<Business Unit/Function>/Dev/<developer name>/lib/programs

Root/<Business Unit/Function>/Dev/<developer name>/lib/outfiles

Root/<Business Unit/Function>/Dev/<developer name>/lib/macros and so on

In QA and PROD, the structure above will be identical except that DEV will be replaced by QA/PROD and the developer name will be dropped.

Based on your experience, would this be a good structure to start with or are we overlooking some pitfalls here with this approach?

Thanks for all your guidance.

Sethu 

[AndrewHowell]

I'd like to ask a couple of clarifying questions:

Are you a business user, an platform admin, business admin?

Your request seem to infer you have scope to make some changes but not across the whole platform.

You seem to have a single platform (nominally Lev1), are you able to install more environments (Lev2, Lev3, etc) on (what would become) a shared platform? You say Licensing prevents you from separating 3 environments, but if it's all on shared infrastructure, does the license matter of it's one Environment or 3 on the same infrastructure.?

@SASKiwi - I concur with your physical & logical separation of SDLC, although I've had conflicts with business & technology over what constitutes "Dev", "QA", etc on a single platform. I've seen several sites where ALL business usage (including their own SDLC) takes place on (what is effectively) the Prod platform, so I tend to use terms/workspaces like "BusDev", "BusOps", etc - which also helps clarify support ownership as well.

[SASKiwi]

You can separate out the three environments by running 3 SASApp server instances on the same physical server. As long as you run them all on the same physical server then the licensing costs are the same regardless of the number of server instances you run. Licensing is based on the number of cores on the physical server.

[SASKiwi · Accepted solution]

Looks like you are heading in the right direction with the folder structures. If there is significant project collaboration happening between your developers then I'd emphasise you should seriously consider using change management tools. These tools provide for both personal and project-based workspaces and simplify your folder management.

Another recommendation is to make use of batch jobs, particularly for PROD and QA as SAS logs a kept by default, monitoring is easy, and scheduling is useful to ensure jobs are run at an optimal time. 

[LinusH]

Given the tools your "developers" are using I would rather think that they are analysts. And then the separation of data/content makes sense. Although, bear in mind that there may be useful fog them to share data/content as well.

"True" developers should /can share data locations and content since their job should be put together in a whole.

[sethuraman_ram]

Thanks @SASKiwi @LinusH and @AndrewHowell for your responses.I think I have enough to begin to set our environments. 

[SASKiwi]

Please mark post as answered in that case. Good luck!