Architecting, installing and maintaining your SAS environment

Using TFS for SAS version control

Reply
PROC Star
Posts: 1,231

Using TFS for SAS version control

I would be happy for any hints/tips on using Team Foundation Server (TFS) for SAS source code management.

I'm just about to start playing with it (don't have access yet).

Happy to hear endorse it in an older thread, I was thinking it might be painful since the main client is Visual Studio.

Assume the simple case of a directory full of .sas files that want to have version controlled.  Do you create a visual studio project that has links to those .sas files (and perhaps other file types, such as compiled formats and templates), and then do check-in and check-out through visual studio?   Or does your SAS code end up living inside a Visual Project file (doesn't seem like that would really be a workable option)?   Or do you perhaps use the TFS windows explorer extension to do check-in and check-outs, like you could with TortoiseSVN?

We will still be developing our code in DM SAS or EG or notepad or whatever, with .sas files sitting on a server.  So curious at a simple level how much you have to change your work flow  as a SAS developer to integrate with TFS. Was at a SAS conference a while back and the pitch for TortoiseSVN was that from the SAS developer perspective, it made version control easy...

Thanks,

--Q.

Valued Guide
Posts: 3,208

Re: Using TFS for SAS version control

Quentin,

You can for a tool but a fool with a tool still is a fool.  You must have some business requirement for this question. What is it?

Common ones:

- Release Management, that is a managed change process of bringing components to a well secure monitored controlled production environment.  (About DTAP)

  This will solve the answer "which version is in production" Release and Deployment Management | IT Process Wiki

- Version Management that one is when having many developers making sure they are not destroying each others work but the programming efforts by them gets managed.

  This is just a more technical requirement for optimizing cost/effectiveness without a direct business requirement where all those version tools are meant for.

The area where al lot of components are getting stored is the SAS metadata.  To manage components stored there you will have first be capable to manage that typical way of SAS storage according to business requirements. As storage normally gets segregated between the DTAP environments (requirement) you should  normally do that wit metadata also.
There are many more storage locations with SAS to be reviewed for that DTAP process SAS flat files (many peoples mind are limited to that) Eguide projects (all included as IDE) SAS Bases catalog types (may be more).

Another common business requirement is having a backup/restore facility. You can backup and restore data from some location and call that version management (what is in a name). 

Ok you can implement a version tool for developers.

- SVN is coming with DI and DI is supporting segregated metadata repositories for developers so you check-in check out the work for them.

- You can store and retrieve classic SAS-code into some VP project.

  You will still needing have the SAS code in a DTAP structure because that code is interpreted and there is no code run-time difference (as is with java).

- what about all other kind of artifacts?  

Where is your global architecture for this?

---->-- ja karman --<-----
PROC Star
Posts: 1,231

Re: Using TFS for SAS version control

Hi Jaap,

Jaap Karman wrote:

Quentin,

You can for a tool but a fool with a tool still is a fool. <snip>

Ummm.. ouch ?

That aside, thanks for your thoughts.

You are correct, I don't have a developed business requirement for the question.  And I would certainly need one if my goal were to evaluate options for version control approaches / life cycle management etc before making a recommendation for some solution (for a group or even just myself).

But for me, my interest in TFS grew out of lunch room conversations with some .NET developer colleagues in another group, who mentioned they were using and liking TFS, and offered to set up a project space for me to play in to see if it might be useful.  And then I was happy to see at least one thread here mentioned that some folks are using TFS with SAS and were happy with it.

My playground business requirement would be: ability to do version control of .sas files during development (check in/out/rollbacks so I don't destroy my own work etc).  So I'm not looking for much.  I would be happy for now with a playground that handles only .sas  files.  Most of my projects are actually heavy on .sas code and light on BI metadata (a few stored processes with some prompts, but source code in .sas files, a DI job, but source code in .sas files).  So even if I had version control only on .sas files during development that would be fun to play with.  If it could handle catalogs, itemstores, etc, that'd be nice too.  And if I wanted to version control the STP metadata , I suppose I could always export it then check it in.  Not worrying about release management, as that is handled separately.

Obviously just throwing out ideas without thinking them through much.  Not trying to address global architecture/enterprise solutions.  But since I know you feel like I go off on my own too much rather than relying on centrally managed solutions from administrators, know that TFS is exactly that.    So it's not like I've decided to develop my own personal version-control solution.  Smiley Happy

Super User
Posts: 3,106

Re: Using TFS for SAS version control

Quentin, we started using TFS/Visual Studio because it integrates very well with SAS and also it is the enterprise application management tool of choice where I work.

The nice thing about TFS is you can use as few of the features or as many as you like. I would recommend starting out just using it for SAS program version control.

One neat feature of Visual Studio is you can configure it to open .SAS files with the SAS development tool of your choice like EG, or Display Manager, or even a text editor if you want.

I use EG. When I want to work on a SAS program I go into Visual Studio, I check out the program so it can be updated, then I double click on the program and it automatically opens in EG. You change the program, save it, then switch back to VS and check the program back in again. It is a very efficient way of working as you don't have to navigate around in Windows Explorer to find your programs or even use EG most recent programs, just go straight into VS and double click on the program you want. Easy peasy.

PROC Star
Posts: 1,231

Re: Using TFS for SAS version control

Thanks much , I didn't realize Visual Studio could work that way.  Definitely sounds like a useful work flow. 

Does Visual Studio / TFS also help with the "build" process for SAS projects?  That is, as points out, seems like I would want my production SAS code to be stored in some directory that is accessible to my BI server, but not writeable by developers.  And then in VS/TFS if I click "build", instead of the usual non-SAS compile to an executable, "build" would mean copy all the .sas files for this project out to that directory.  Is it something like that?

Super User
Posts: 3,106

Re: Using TFS for SAS version control

Yes, it can be done. Our TFS guru has written a custom script that will build a SAS deployment automatically and write it out to a production folder location. Behind the scenes this is simply a file and directory copy of the latest version of all of the programs. The steps to do this are:

1) Check in all programs required for a deployment.

2) Create a label for the deployment which can be just a version number like Release 1.2.3, and apply it to the top folder of your SAS program folder (aka directory) hierarchy in VS.

3) Run the custom SAS deployment script referencing the created label, and it will automatically copy it out to the correct production location. You can tell TFS to permanently store the deployment so that you can keep a full history of them if you want. At any time you can go back to a previous deployment and put that back into production.

Also you might be interested to know that the VS program version compare does a very good job of comparing SAS program versions (what has been added, deleted or changed is highlighted in different colours). It has saved my bacon on a few occasions where I have accidentally deleted some code, which I was able to easily identify via the version compare and put it back in again. 

Valued Guide
Posts: 3,208

Re: Using TFS for SAS version control

SASkiwi interesting as you focussing on classic text like code. May I add some complication with that?

1/ Adding SAS formats (or FCMP procs) into a project. These are needed as compiled versions stored in SAS catalogs.

    As far as I see it will add a fixed config setting belonging to that sas-application build environmnet. (autoexec/config)

    This concept failed to implement with Endevor (CA mainframe) a long time ago

2/ The classic old style Text editing approach is well covered with those generic tools.

    It is one of the arguments WPS is using or can use as SAS competitor.

    There is a lot that not like that:

    - the SAS Metadata structure with all those artifacts and a folder (storage)

    - Eguide tasks builder and flows

    - schedule processing (grid)

    - Web-containers with all stored data (SAS VA and aal metadata)

    What about those?

Are you sticked to old classic coders or open for new approaches?

When you are open for new approaches how would version (and release) management fit into those.       

---->-- ja karman --<-----
Super User
Posts: 3,106

Re: Using TFS for SAS version control

Definitely open to new approaches Jaap. We have no need to version control SAS format catalogs as they are completely built from versioned source code. Our SAS Metadata only contains users and data sources so the SAS and server backups are sufficient to manage those. We don't use EG tasks/projects for our production applications, as they must be easily reviewable and auditable. We also use batch processing a lot so again SAS code is easier for this. SAS macro is the way we improve productivity.

BTW we have been trying SAS Studio and are really impressed by it, and it also works well with TFS. By sticking with SAS code it is easy to switch between EG and SAS Studio and it all works exactly the same.    

Super User
Posts: 3,106

Re: Using TFS for SAS version control

Definitely open to new approaches Jaap. We have no need to version control SAS format catalogs as they are completely built from versioned source code. Our SAS Metadata only contains users and data sources so the SAS and server backups are sufficient to manage those. We don't use EG tasks/projects for our production applications, as they must be easily reviewable and auditable. We also use batch processing a lot so again SAS code is easier for this. SAS macro is the way we improve productivity.

BTW we have been trying SAS Studio and are really impressed by it, and it also works well with TFS. By sticking with SAS code it is easy to switch between EG and SAS Studio and it all works exactly the same.    

Valued Guide
Posts: 3,208

Re: Using TFS for SAS version control

Ok Ian, I am getting to see where the difference insight is coming from.

You are right when sticking to those classic coding approach (.sas files) you can put those in the classic code versions tools. That is like using it in the Cobol/Fortran way.
For ETL there is that DI studio with cubes as possible goal and getting with into some others like information map studio. The data sources are extended to record/fields as becoming the real source with transformations jobs. With SAS-VA that part is even going far further. These are the parts and area-s SAS is pushing you should use. That is leaving that old classic code building behind you.   

---->-- ja karman --<-----
Super User
Posts: 3,106

Re: Using TFS for SAS version control

DI Studio is on our radar but is too expensive for our budget and we are not yet an enterprise platform where it might be justified. SAS VA is also on our radar and I would love to try it out, but we have more important fish to fry right now....

PROC Star
Posts: 1,231

Re: Using TFS for SAS version control

Hi @SASKiwi et al.  I have a followup question (should probably ask in a TFS forum, but thought I'd try here since it's probably a friendlier group Smiley Happy

 

I have .sas files on a linux BI server which are invoked when a stored process runs.  The linux directory is available to windows via SAMBA, and amazingly, it was pretty easy to get all these files checked into TFS.  And TFS seems to be happy managing linux read/write permissions on the files, even though it's going through SAMBA.

 

The only disappointment that in my current setup I have a /DEV directory structure and a /PROD directory structure.  And I checked the /PROD directory into TFS so it is my local repository.  And when I want to promote code, I copy it from /DEV to /PROD.   (Obviously this is not the way TFS expects to be working, but I think it's the best I can do since the prod code is live code sitting on a BI server).  The disappointment is that even when myprog.sas is checked in to TFS and is read-only, I can still overwrite it by using OS commands to copy /DEV/mpyrog.sas to /TEST/myprog.sas.   And then at that point my TFS repository is out of synch with my local repository ( / PROD on the BI server), and I'm out of luck.  That is, TFS doesn't even know that my /PROD code has been changed.

 

I think the real issue may be that on linux, even when myprog.sas is read only I can over-write it, and TFS isn't managing the permissions for directories  (I haven't tested if TFS could handle this like I would want on Windows).  For TFS to work like I would want, I think it would need to make the directory /PROD be read only.   I think if the directory were read only, I would NOT be able to overwrite files in the directory.  And when I check out a file, it would need to make the directory be writeable.  (Ugh, even then it wouldn't work really, because if I checked out a single file and the directory became writeable, all files in the directory could be over written.  So really it would mean I would need to be able to do check-out and check-in for all files in a directory at once, instead of the file-level, which doesn't sound good....)

 

So, is there a way to make TFS manage directory permissions for the local repository in the same way as it manages file permissions?  Or some other approach?

Super User
Posts: 3,106

Re: Using TFS for SAS version control

Hi @Quentin. I use TFS exclusively under Windows. All TFS Workspaces - the folders and files under TFS control - are locked down and you can't change them via the operating system. In Windows Explorer you see a "green corner" on the folder icons under TFS control that highlights the fact you can't change it outside of TFS. In WE you can actually go into folder/file Properties and do TFS actions on a folder or file which is kind of cool.

 

It sounds like under Linux this type of control may not be possible. I suggest you Google TFS Linux workspace to see what others have done. One thought - you could perhaps move your main "working" TFS workspace to Windows, then deploy separately to your /dev and /prod folders. You can then use OS permissions to restrict access to these.

PROC Star
Posts: 1,231

Re: Using TFS for SAS version control

Thanks much @SASKiwi, appreciate the quick response and thoughts, that's a big help.

 

I have a "green corner" on the folder icons and files, which I took to mean "checked in."  If I check out a file, it becomes writeable (green corner with pencil), and when I check in it goes back to green and read only.

 

I looked a little bit more into permisions on my directory accesible via Samba.  The Win 7 folder properties knows the files in it are read only.  But as I go into the security tab and advanced, Windows says I have "special permissions" which give me full control of the folder itself to add/delete files etc.  Which is what I don't want, but I think is an accurate reflection of my permissions coming from Linux.

 

I think I'll play a bit with setting up a TFS workspace on Windows, just to see how its permissions look different from my workspace on Linux.  Then will do more googling or try s.o.  Agree, the real answer is for me to write a real deployment script, rather than trusting my fingers to copy a file or two  from /Dev to /Test each time I need to make a little tweak...

 

I can't really put my TFS workspace on Windows, because without a Windows BI server, I wouldn't be able to test the code from a browser (it's stored processes called from Web).  I suppose in theory I could mount the windows directory to Linux, but that feels a bit dirty...

 

If you're heading to Vegas next week for SGF please let me know.  I'll buy you whatever drink you choose (and pester you with more questions as you drink it.  : )

Super User
Posts: 3,106

Re: Using TFS for SAS version control

Sorry but I'm not headed to Vegas next week, but having never been to Global Forum maybe I should check it out some time!

 

The more I use TFS and SAS together the more I think they integrate so well. Please explore the use of deployment scripts. They work brilliantly for us and allow us to keep a saved copy of every Production deployment should we ever want to go back and use it.

 

Our applications get audited a lot and having all SAS code under change management control is a big tick in the box for us.

 

Enjoy Vegas! 

Ask a Question
Discussion stats
  • 14 replies
  • 1919 views
  • 4 likes
  • 3 in conversation