Git in combination with code executed on SAS server

[NicoM]

I am working in an environment with SAS code running calculations that are executed on a SAS server. Running code on our laptops is not possible as the laptop only has SAS clients installed (EG and Mgmt Console). At the moment we are doing the version control of the SAS code on the server manually. This can be quite cumbersome with multiple people making changes on the code. Luckily, we are a small team and it is still manageable.

Despite this, we would like to switch to Git. One of the advantages we see is the ability to create branches, which enables developers to work in parallel on different branches. I am used to working with Git in an environment where you can execute code locally on your laptop (e.g., with Python). This way you are completely independent of others.

What is the best way to set this up in an environment with a SAS server that all developers are dependent on to execute the code? The only thing I can think of is that we create a separate code folder for each developer. Each developer can checkout a local copy of the Git repository to this folder to make changes. This would require some modifications to the paths and SASAUTOS during initialization, but it is manageable.

Any alternative suggestions or other things we should consider with such a setup?

[FK1]

Hi @NicoM ,

although I do not completeley understand your comments regarding the environment you run your SAS Code in (remotely on a server versus locally on your Laptop), I wanted to say, that it is possible to use your local SAS Client, e.g. EG on your laptop, in order to run SAS code remotly on your SAS server. Usually this is done by establishing a connection to a Metadata Server.

Meaning, the fact, that you "only" have a client installation on your local laptop does not imply, that you are unable to run SAS Code remotly on your SAS server.

When it comes to version control, EG offers some built in functionalities, however, theses are limited in comparison to the CLI possibilities of GIT. Depending on your version of SAS, you could also use the available SAS Git functions.

Maybe you can also find some interesting suggestions at this years SGF:

Git and SAS®: A Match Made in (SAS®) Studio

Thursday, May 20, 4:15 PM-4:45 PM GMT+2

[NicoM]

@FK1, thanks for your answer. The comment regarding the laptop was simply to clarify that running the code locally is not an option. This would be the easiest solution when working with Git.

Since all developers are dependent on the SAS server to run the code, I think we need to have the "local" Git repository of each developer on the SAS server. Is this indeed the only way?

By the way, the main/remote Git repository is in Bitbucket.

We indeed connect to a metadata server using SAS EG to execute the code on the SAS server.

[FK1]

Hi @NicoM ,

I think you are right. At least, I also cannot think of another way than for every developer to have his own "local" Git repo on the SAS server....

Maybe someone else in the community might have some other ideas?

[Sajid01]

Hello @NicoM 
You have two questions
(1) How to create a local repository on each developers laptop/desktop.
  In my opinion the simplest approach is to use git-bash. You can also use git function, source tree etc. It is partly a matter of choice and partly company policy/plan. As  you are using Bitbucket SourceTree may be the option the developers may get.

(2) I think we need to have the "local" Git repository of each developer on the SAS server.
No not needed.  The production SAS server (or simply SAS server if you have only one), may have a local copy of the master branch  (or by what so ever name it may be called)..

(3) As you are using Bitbucket all the branches will be on the Bitbucket server (either on premise or in cloud)  and the production server may be having a local copy of the master branch in a typical scenario.
Lastly on a lighter vein  whatever the developer may plan, it is the devops team that calls the shots.

[NicoM]

@Sajid01, we can create the local repository with several tools. I have TortoiseGit, SourceTree, command line and EG as options. However, there is no point in creating a local repository on our laptops. As I mentioned in my initial question, we cannot execute any SAS code on our laptops. The laptops only have SAS EG installed, nothing to run any code. That should be done on the (Development) server.

I agree that the Production server would only have one version: the latest version of the master branch.

[SASKiwi]

@NicoM  - Interesting post. We are currently investigating a very similar solution. We are migrating from using MS Visual Studio with Team Foundation Server (TFS) to MS Visual Studio using Azure DevOps which is essentially Microsoft's implementation of GIT. 

MS Visual Studio is definitely worth checking out as a version control tool for SAS programs. It works very well with SAS Enterprise Guide. We haven't yet joined the dots on GIT best practice though. Happy to update you once we've figured it out though. Our experimentation to date indicates that having individual cloned GIT repositories for each SAS user stored on our SAS App server is definitely a workable solution. Via folder shares EG can open SAS programs from each repo and using %INCLUDE to bring in other programs and using AUTOCALL macro libraries from these is entirely possible.   

[NicoM · Accepted solution]

@SASKiwi, thanks for you reply. This sounds very similar indeed. In the meantime, we have indeed also applied the practice of having individual "local" repositories on Development SASApp server for every developer. This works fine. We have created a single path parameter (global macro variable) that points to a specific folder to make sure the correct folder is looked into for the code. This parameter is used in includes and setting SASAUTOS.

We have also kept the main/non-user specific folder for the code; i.e., where our code was located before we started using Git. We decided to do this as on Acceptance and Production the code will still run from that same location.

We did turn it into a Git repository as well and it seems to be working fine if multiple people pull updates from the master branch into this folder.

The only thing I am still struggling with is the versioning of the SAS metadata. We do not rely on it too much, but we have stored processes and table definitions in the metadata that we rely on. It would be nice to somehow add Git versioning for these as well.

[SASKiwi]

@NicoM - Thanks for the feedback. You are indeed using Git in a very similar way to us. We are also using the Pipeline feature deploy code and other application  artifacts into production.

Regarding SAS metadata - IMHO this too volatile and complex to put under version control. We rely on the daily metadata backups if we need to revert to an earlier version.  

[NicoM]

@SASKiwi, I agree that it is not easy to put metadata under version control indeed. Sometimes we do have changes in the SAS metadata that should be deployed together with the modifications in the code. For example, changes to (parameters in) stored processes and/or table definitions on which the code relies.

This means that once these updates to the code make it into a higher environment (e.g. Production) these changes to the metadata need to be installed there as well.

Updating the code on a higher environment would then simply mean pulling the new commits to the master branch into the local repository on Production. For the metadata, I am thinking of creating a snapshot of the metadata (SPK file) on the Test server once we are happy with the release branch (right before merging it into master). This SPK file can then be imported into the metadata at the same time as the changes to the master branch are pulled into the repository on Acceptance or Production.

By the way, does the "... other application artifacts ..." you mention in the context of this "Pipeline feature"  include SAS metadata? That is, do you use this feature to automate the promotion of your SAS metadata as well?

[SASKiwi]

@NicoM - The other application artifacts are typically Excel spreadsheets or SAS datasets containing non-volatile reference data. No metadata is included. If it did then it would have to be in the form of SPK files like you mentioned. Luckily we only use metadata for defining data sources and user-related purposes and not for applications like stored processes. So really there is no significant business need to include it.

We are also mindful that we will have to migrate to SAS Viya in the next few years (since SAS 9 is going to limited support beyond 2025) where there is no SAS metadata like in SAS 9. How we integrate Viya and Git, we have yet to get our heads around. It's a problem for another day... 

[ChrisNZ]

Edited. Never mind 🙂

[AllanBowe]

We (the SASjs team behind Data Controller) have implemented an approach where each developer can work locally (using VS code).  Developers work in their branches, can run git commands, no need to grant SSH access to a production server. 

Just execute the code in SAS when you need to run it.  All code is deployed as either SAS 9 STPs or Viya Jobs, so can be executed from command line or the VS Code extension.

The framework we built includes HTML documentation (using Doxygen) and a unit testing facility.

More info:

• https://communities.sas.com/t5/SAS-Communities-Library/DevOps-with-VS-Code-GIT-and-SASjs/ta-p/747648
• https://communities.sas.com/t5/SAS-Global-Forum-Proceedings/Scaffolding-SAS-Projects-With-NPM-and-SA...

[NicoM]

@AllanBowe, interesting post. This would be the ideal way of working with Git indeed. In the article of the first link you mention in step 3:

SASjs does NOT depend on a server filesystem. SASjs does NOT require catalogs, nor SASAUTOS configuration. Instead, every Job / Service / Test has dependencies (eg macros or freeform SAS program includes) defined in the header.

I assume that this also means that existing projects, that do depend on SASAUTOS and the SAS server file system, need to be restructured in order for this to work? Something which I would definitely consider doing, by the way.

[AllanBowe]

it works for us - all our projects are completely inside GIT, and the nice thing is that you can deploy to ANY sas environment, no dependency on an existing configuration / setup.

About the restructure - yes indeed.  Of course, ultimately you are just deploying SAS code so you CAN continue to use your existing autoexec / SASAUTOs / catalogs / filesystem / whatever, however - it's no longer necessary and we think it's better practice to keep everything in GIT.