Using TFS & Release Management for source control & automated deployments

[AllanBowe]

After building /  deploying a successful release deployment system as an STP web app using SVN, following the excellent design principles here, I have now been tasked with re-wiring everything into TFS.

We have TFS 2013 (so, seperate RM client), SAS 9.3 and four Windows SAS environments (DEV, TEST1,TEST2, LIVE).  Most of our work is DI ETL, scheduled in LSF.

Whilst I know there IS a project that allows GIT merging of SPK files in 9.3, it is subject to certain restrictions that are not palatable for us.  Therefore (like 99% of us) our branching strategy is basically..  no branching.  And we will have the following source control folder structure:

ENVIRONMENT 

- BASELINE

-    /SASApp  (everything under Lev1/SASApp, minus logs etc)

-    /ROOT.war   (css, js, images, html etc)

-    (various other folders in source control)

- RELEASES

-    /Release XXX  (subfolders containing release artefacts)

Developers will have a web front end so they can create a new 'package' (creating a new releases subfolder), and then use a 'picker' to select metadata (jobs, tables, flows) and physical (.sas, .js, .html) files for inclusion in the package.   They will also provide any manual deployment instructions.

All developers work (check in metadata, update .sas files) in the DEV environment.  When they are ready to deploy (to TEST1) they will select the appropriate 'build' in TFS.  A custom build step will execute a powershell script, and I'm hoping it will be possible as part of this process to pass through some kind of release identifier.  The powershell script will call an STP webapp, which will use a system account (sassrv equivalent) to look up the Release folder and perform an automated deployment of all the Release folder contents (.sas, .spk etc) into the target environment.  It will take a backup first (so that the deployment can be undone), then import all the metadata, deploy the jobs, check in the physical files & capture the logs.  Data Model updates are an example of a manual step (for now).

Presuming the build / tests are successful, the package folder contents can follow the same deployment route (via Release Manager) into TEST2, and then into LIVE (with appropriate signoffs in the Release Manager client tool).

My questions were...  What are the pitfalls with this approach?  Could it be done better?  What tips / tools / techniques should I be aware of?  Can anyone share experience / code / diagrams on this subject area?

Our whole IT department uses Visual Studio and TFS, and staying with SVN is no longer an option (despite the handy DI Studio plugin).

[SASKiwi]

Check out this post to which I contributed:

https://communities.sas.com/t5/Administration-and-Deployment/Using-TFS-for-SAS-version-control/m-p/1...

Unfortunately we don't use DI Studio so I can't really offer any insights using it with TFS. All I can say is that TFS integrates very well with SAS particularly if you use mostly SAS code. Also the automated scripting of deployments is great, along with the ability to save all deployments. 

[AllanBowe]

Thanks @SASKiwi - good to know the setup can be done succesfully.  I guess that working purely with .sas (text) files is relatively straightforward, as it permits all kind of branching / merging strategies and deployment is a simple copy / paste (or checkin / merge) job.

Binary files (.egp, .spk) are more of a challenge.  Did you have a TFS build server on your SAS application server?  Or as a separate instance?  Was your workspace on a local desktop?  We don't have Base SAS licences, so we do all of our development directly on a shared DEV BI server.  The TFS concepts of continuous integration, builds etc lead me to believe that we should use a shared Team Explorer workspace (on the DEV server) and do our builds against a separate (TEST1) SAS server.

How do you identify the .sas files you want to promote?  Do you label them during checkin?  Or do you promote a certain branch?  Are those .sas files stored in a separate (releases) folder?

Currently on pluralsight trying to get up to speed on TFS & release management!  

[SASKiwi]

In our case TFS and SAS are completely separate. Each SAS user has their own personal copy of the TFS repository which is displayed in MS Visual Studio. Text (SAS programs) or Non-text (EG projects, SPK) files can be easily dropped into a TFS folder by dragging and dropping from Windows Explorer. Double-clicking on an artifact in a TFS folder will open it in the tool of your choice. SAS programs and EG projects will open in EG for example.

We have a shared Team workspace for all of our SAS development. We also have a custom deployment script which runs on this workspace when doing a TFS build. It copies files to our Prod SAS environment into the correct folders automatically. A complete history of every build is kept along with the build artifacts.

We deploy all latest versions of artifacts held in a particular workspace folder. You can do stuff by labels as well but we wanted to keep it simple. If certain programs aren't being productionized we keep them in separate non-deployed folders.