SAS Studio DevOps: Collaborative Development of SAS Studio Flows and Programs Using Git Integration
As we jump in to see how this works, let me outline a very simple scenario as context for this work. Suppose this SAS Studio flow shown below is maintained by a user I'll call Alex. It joins two source tables to create a target table called OrderAnalysis.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Suppose also there is a related program, shown below, for which a user called Fiona is responsible. It uses the OrderAnalysis table produced by Alex's flow as source table to generates summary reports.
The current flow and program are in production as a project we will call ETL and Reporting and the project is managed by Sarah. Any updates by either Alex or Fiona to the individual components they are responsible must not affect their dependencies in the current production version. Any changes to the flow or the program need to be tested to verify that no dependencies have been compromised. It is Sarah's responsibility to push changes to the production project once the tests have been verified.
Using SAS Studio's Git integration Alex and Fiona can independently update development versions of each of their content pieces and conduct tests to confirm that no project dependencies have been disrupted, all without impacting the production project. Sarah can then push updates to the production project with a simple push of a button. Here I will step through how Sarah, Alex, and Fiona can collaborate to accomplish this.
Step 1: Sarah creates a GitLab project
To begin, Sarah will create an account at gitlab.com, if she does not already have one. It's easy and free. GitLab will be used for this scenario, but SAS Studio works with other Git servers including GitHub, BitBucket and others in the same way. Once logged in to her GitLab account, she can click the New project button on her account home page and select Create a blank project. Let's call this project Production. She can leave defaults for the rest of the options and click Create project.
The project called Production with a single main branch is added on the GitLab host with nothing in it but a readme file. Sarah can now clone this project to a local SAS Studio Git repository, copy the existing ETL and Reporting production content to the local repository and then push the content to the remote repository. To prepare for that Sarah will gather two pieces of information. She will need the URL for the Production repository stored on the Git server and she will need to create and copy access tokens to allow access to the repository. To find the URL, (1) click Code in the Production project and (2) copy the value in the Clone with HTTPS field.
Note: For this post, HTTPS is used for cloning the repository to SAS Studio. SSH is also supported.
Step 2: Sarah creates two Access Tokens for the project
Next, Sarah will create two access tokens. The first will be used by Sarah herself and defined to allow her full access to push updated to the main branch of the Production repository. The second access token will be used by Alex and Fiona and will be defined for read access to the main branch and full access to a other branches of the Production repository. Alex and Fiona will be able to add new features in a feature branch and only once tested and approved, Sarah will then be able to push those updates from the feature branch to the main branch.
In the left side menu in GitLab she selects (1) Settings and then(2) Access Tokens.
In the Project Token Access screen (not shown), she clicks the Add new token button. She'll chose Ownertoken as the name, an expiration date sometime in the future, and Owner will be selected for the role. This will be the access token Sarah will use to connect from SAS Studio and the Owner role will allow her to push changes in the SAS Studio content to the main branch of the repository. She will also select api under Selected Scopes.
Next she click Create project access token at the bottom of the screen. At this point it is necessary to click the copy icon to copy the access token and save it for later use in SAS Studio. It will not be accessible for copying once you leave this screen.
Sarah clicks Add new token again a second time, this time creating a second token that she names DeveloperToken, assigns an expiry date in the future, a role of Developer, and api as the selected scope. She copies the access token as before and save to the text file. Alex and Fiano will access the project with the Developer role and thus be able to create feature branches and push changes to those branches, but they won't be able to push changes themselves to the main branch.
NOTE: For each user that will access the repository on the Git server Sarah will also need to add them, or a group they are part of, to the project membership. For details click here.
Step 3: Sarah clones the GitLab project to a SAS Studio folder and adds SAS Studio content to the project
Back in SAS Studio, Sarah creates an empty folder and clone the Gitlab Production repository to that empty folder. Git repositories can only be cloned to empty folders. Once the folder is cloned a a local repository, Sarah can copy the project content to the repository folder. Let's call the folder called aa_Production.
Sarah will (1) click the Git Repositories section of the navigation pane and then (2) click Clone Repository.
She copies the Repository URL that was copied earlier in step 1 above into the Repository field and for Server location clicks the folder icon and navigates to the aa_Production folder. In the Profile Create new is selected for the Profile
Once Create new is selected a second window opens to define a profile. Sarah will select the HTTPS tab as the method type RepoOwner for the profile name. The username and email corresponds to a GitLab.com user account and she enters her own. Users will require a Gitlab account to access the repository.
When Sarah clicks OK, the repository shows in the Git Repositories section.
In the Explorer window, the aa_Production folder is displayed with a repository icon. You may need to refresh a higher level folder above aa_Production to see the icon change.
Sarah can now copy the contents of aa_Finalized into the aa_Production repository folder, return to the Repositories section of the navigate pane, double click the aa_Production repository to open it up and see in the Commit tab the ETLFlow and the OrderAnalysis SAS program that have been staged in the local repository. To push these changes to the remote repository, Sarah can (1) mark all the files as staged, (2) add a commit message, (3) click Commit, then (4) Push the changes to the main branch of the remote repository.
The project at gitlab.com now shows the newly pushed content.
Step 4: Sarah creates a development branch in the Git project
Sarah will now add a development branch to the repository called featuredev for Alex and Fiona to use. Their updates will be stored separate from the main production branch. To do this, Sarah can navigate to the Git Repositories section in SAS Studio, open the aa_Production repository, click the History tab, right-click on one of the commit messages, and select Create new branch.
In a subsequent screen (not shown) she types featuredev as the branch name. The new branch is added and by default is set as the current branch. Any changes to repository content will be made in the branch that is current.
Note: At any time, you can click Current branch and then select a different branch to make it the current branch:
Before handing over the development work in the repository to Alex and Fiona, Sarah will update the code in the project that defines the library location for the target data produced in the project because any changes to the target table created by the ETL flow during development must not impact the target table used in the production application. So, Sarah will add a Server folder in SAS Studio called aa_TargetDev and she will add some macro code to the SAS Program step in the flow and to the OrderAnalysis program in place of the original LIBNAME statement for the target library.
This program will point the target library to the aa_TargetDev folder when the user logged into SAS studio is Alex, Fiona, or Sarah. Otherwise the aa_TargetProd production folder is used for the target library. (NOTE: Alex, Fiona, and Sarah will also now each need a separate login account for SAS Studio if they need to use the production project content).
To commit and push the featuredev branch and it's updated content to the remote repository, Sarah will click the Commit tab in the aa_Production repository, and, as described earlier she will (1) mark all the files as staged, (2) add a commit message, (3) click Commit, then (4) Push the changes to the featuredev branch of the remote repository.
Even if Sarah did not have a need to update the target library code, it is necessary for Sarah to push the newly created featuredev branch to the remote repository. Otherwise the featuredev branch will not exist in the remote repository and Alex and Fiona will not be able to access it when they connect to the aa_Production repository.
Sarah's work is done for now.
Step 5: Alex and Fiona clone the GitLab project to SAS Studio and checkout the featuredev branch
Alex and Fiona can now each login to SAS Studio, each create a folder called (Alex could call his aa_DevAlex and Fiona could call hers aa_Fiona), and each of them use the same method Sarah did to clone the aa_Production repository, except they will clone them to their own unique folder, use their own gitlab account name and email in their Git Profile, and they will both use the DeveloperToken access token for the password (they can get that token from Sarah).
To checkout the featuredev branch, they will then each access the Git Repositories section in SAS Studio from the Navigate pane and open their Git repository. In the screen capture below for Alex, he has opened his aa_DevAlex repository. Initially main will be the active branch. To active the featuredev branch, Alex will (1) click the History tab, right click on the commit message that shows the origin/featuredev branch, and then (3) select Checkout branch --> origin/featuredev.
Step 6: Alex and Fiona develop independently in their own local clones of the featuredev repository and push changes to the remote featuredev
Alex and Fiona are free to develop and test each of their project components in their own separate local repository clones of the original content. When they have completed updates they can use the Commit tab as describe earlier to mark files as staged, commit them and push them. The push will go to the featuredev branch in the remote repository on the Git server. The Git server will thus contain all changes pushed by both Alex and Fiona.
Note: If both developers make changes to the same file the Git server will detect conflicts when the second user attempts to push. The conflict will need to be resolved before the push can proceed.
Step 7: Sarah pulls completed changes in the featuredev branch to her local repository and tests
Once all updates by Alex and Fiona are ready to review Sarah can update the featuredev branch in her local repository by opening her aa_Production repository, making sure that the featuredev branch is the current branch and clicking on Pull featuredev.
The updated content is now pulled into the SAS Studio environment and Sarah can proceed to execute the flow and the SAS program and verify that their are no errors and results are as expected.
Step 8: Sarah pushes changes to the main branch
Sarah is ready to push the updates to the main branch which stores the production content. To do this in SAS Studio she (1) clicks on Current branch and (2) selects main to make the main branch the current repository.
Next, she will (1) click the History tab, (2) right-click the most recent commit message for the featuredev branch, and (3) select Merge into main --> featuredev.
This merges the updates into the main branch of the local repository. This updates the production content in SAS Studio for all users of the project. The main branch of the remote repository also needs to be updated to be in sync with the current production content in SAS Viya. Sarah can accomplish that by clicking Push main.
Note: As alternative method for step 8, Sarah can login to her account at gitlab.com and use the tools in the gitlab.com project interface to push the featuredev branch in the project on the Git server to the main branch. Subsequently, Sarah would return to SAS Studio and click on Pull main to pull the changes from the remote main branch to the local main branch.
Additional resources
- Youtube: Git and SAS: A Match Made in (SAS) Studio
- SAS Documentation: Understanding Git Integration in SAS Studio
- SAS Communities Library: SAS Explore 2023 – SAS Studio Git Integration Hands-On Session
Find more articles from SAS Global Enablement and Learning here.
