DevOps with VS Code, GIT, and SASjs

[#1]

According to github data, Visual Studio Code is the third most popular IDE in the world - whilst this linkedin poll shows that  90% of SAS developers actually prefer  Base SAS, Enterprise Guide or SAS Studio.  This is a true testament to the great work by SAS R&D, supported by the fact that third party editors like VS Code do not ship with any SAS functionality.  There is a VS Code marketplace though, and over the last 12 months my colleagues in the SAS Apps team at Analytium have been hard at work building an open source framework for SAS DevOps that fits nicely into an extension.  For more information on how this framework (SASjs) can accelerate the development, deployment, quality checking, testing and documentation for Data Engineering projects - read on!

But first - why VS Code?  We say it is a great choice of IDE for SAS Developers, for the following reasons:

• Ability to run terminal commands
• Vast marketplace with third party extensions
• Ability to compile and execute SAS projects locally, without impacting other developers.

This last point is key for those with a background in GIT workflows.  The traditional way to use GIT in software development, is for each developer to have their own local clone of a shared code repository - where they can:

1. Make a feature branch
2. Build / Deploy / Test changes
3. Apply standardised code styles (linting)
4. Commit changes (perhaps using the conventional commit standard)
5. Create a Pull Request (also known as a Merge Request)

At this point the change is reviewed, and - assuming it looks good and the tests are passing - the green button is clicked and an automated pipeline will deploy the project to the server and update the documentation.

There are many flavours / variations of / extensions to the above, but this is the general approach.  So how can this work in VS Code, on a personal laptop?  Let's work it through with an example!

Step 1 - Prerequisites

The following software will need to be installed locally in order to proceed:

• VS Code - the IDE we will work with.  Portable version available.
• NodeJS - the run time for SASjs. Portable Guide available.
• GIT Bash - choose this as your default terminal shell in VS Code.  Portable Guide available.
• Doxygen.  Used to produce the HTML documentation for your SAS project.

NodeJS (NPM), GIT, and doxygen will all need to be in your PATH.  You may need to restart VS Code to see it take effect there.  On some windows machines, you may need to open the doxygen executable before it can be 'seen'.

To check everything works, you should be able to run the following commands from the terminal:

npm -v
git --version
doxygen -v

Make sure you are on NPM v7 by running the below (or use NVM):

npm i -g npm@latest

You can now install the SASjs CLI as follows:

npm i -g @sasjs/cli

Right - ready to rock and roll!

Step 2 - Environment Setup

Whilst SASjs was initially conceived to support the development of SAS Apps (like Data Controller), the functionality that was created to automate the backend (SAS) part is highly applicable to pure Data Engineering / Data Science projects.  So we will focus on a SAS only project in this example.

There are two ways to obtain the seed apps:

# Option 1 - clone the repo
mkdir myproject
cd myproject
git clone https://github.com/sasjs/template_jobs.git .
rm -rf .git
git init

# Option 2 - use the sasjs create command
sasjs create myproject -t jobs
cd myproject
git init 

Now we have an environment, we need to connect it to SAS!  The easiest way to do this is by registering a client and secret.  You'll need an administrator for that - see here for options.  Remember to request the 'authorization_code' grant type - don't be embedding passwords in your applications.

Once you have a client/secret you can run the following command, and follow the prompts to authenticate and obtain the access / refresh tokens.

sasjs add cred -t viya

Be sure to choose the 'VIYA' serverType, provide the URL of your Viya server, and choose the LOCAL target option.  More details here.

The final thing to configure is the appLoc.  This is the location (eg metadata folder in SAS 9, or SAS Drive folder in Viya) under which the project will be deployed.  The appLoc is adjusted by modifying the sasjs/sasjsconfig.json file.

Step 3 - Compile, Build and Deploy the Project

For this section it's worth pointing out a premise of the SASjs framework.  The source of a project, is the GIT repository!  Which translates to your local branch of that repository.  SASjs does NOT depend on a server filesystem.  SASjs does NOT require catalogs, nor SASAUTOS configuration.  Instead, every Job / Service / Test has dependencies (eg macros or freeform SAS program includes) defined in the header.  During the compilation process, each dependency is bundled such that each final Job / Service / Test is fully self contained (in the STP metadata or Viya Job definition).   To repeat - there is no dependency on the physical server filesystem!

After all artefacts are Compiled (in the local sasjsbuild folder), the Build process creates a single JSON file ready to deploy.  The Deploy process takes that JSON and creates all the artefacts in Viya, split over three SAS Drive folders under the appLoc:

1. /jobs/ - all folders specified in the `jobFolders` array of the `jobConfig` object in the sasjsconfig.json file
2. /services/ - all folders specified in the `serviceFolders` array of the `serviceConfig` object in the sasjsconfig
3. /tests/ - split between macros, services & jobs for folders specified in the `testFolders` array in the `testConfig` object

You can compile, build AND deploy your jobs, services and tests in one command:

sasjs cbd

The same project can be compiled / built / deployed in different ways to different targets.  A target is comprised of:

1. Target Name (alias for the target)
2. Server Type (eg SAS9 or VIYA)
3. Server URL (including port)
4. App Location (location in the folder tree)

By creating a target for each developer (unique server App Location) it is possible to build & deploy in isolation until features are ready to be merged to a shared branch.  No more tripping on each others toes!

For more info, see docs for compile, build and deploy.

Step 4 - Run Tests

SASjs enables unit testing of SAS Jobs, Services and Macros by simply creating a file with the same name, but with a ".test.sas" extension.  Each test runs in it's own isolated session (Stored Process or Viya job) and returns results in JSON format - you can write this output yourself, or use one of the existing assertion macros. 

The process is as follows:

1. Write your test
2. Compile / build/ deploy it (`sasjs cbd`).  At this point you will see the code coverage.
3. Execute the tests.  At this point you will get the test results.

To execute the tests, simply run:

sasjs test

Results will be returned in CSV, JSON and JUnit XML format.  More info in the docs.

Step 5 - Commit the changes

Wait - just a moment.  Before you commit.  Are you sure you followed that style guide?  Removed all that white space?  Followed the correct indentation?  Removed any encoded passwords?

Thanks to sasjs lint, you don't need to actually remember to do this.  Just configure the settings you need, and add the "sasjs lint" command to your pre-commit git hook.  This will run the check before every commit, ensuring that standards are always followed, and reducing the time spent on 'nit picks' during the code review stage!

 If you are using the SASjs extension, many problems (such as a missing doxygen header) can be fixed by right clicking and selecting 'Format Document'.

 The warnings are also shown in the 'Problems' tab of the terminal window in VS Code. 

Step 6 - Update the Documentation

By following the doxygen standard for program headers, it becomes trivial to generate documentation directly from SAS programs.  SASjs also enables the generation of Data Lineage, by simply defining the Data Inputs and Data Outputs inside each job.

The homepage for the docs is take from the project README.md but you can change this using the docConfig settings.  

To generate the documentation, simply run:

sasjs doc

 The output will go in the `sasjsbuild` folder by default.  The destination can be changed with the outDirectory property.  An example of documentation generated in this fashion can be seen at https://core.sasjs.io.

For more information on sasjs doc, see the doc documentation!

Step 7 - Iterate

At this point there will be a repeat cycle of making changes, testing them, documenting, merging, building automated release pipelines, and delivering business value.  Perhaps you will work on CI/CD pipelines for running individual jobs and flows.  Perhaps you want to run SAS code directly from VS Code.  The following commands may help:

Command	Description
sasjs run	Execute arbitrary code from anywhere
sasjs job execute	Trigger a viya job and wait for the result / log
sasjs flow	Run jobs in waves with dependencies, fetch the logs, stop on error or warning
sasjs folder	Manage SAS Drive folders from commandline

More commands are listed here and further resources are listed here.

If you find value in these repositories, don't forget to leave a STAR, or engage with our developers in the discussion boards.  We welcome contributors!  If you like what you see, but you just need some particular feature, then - like any other open source repository - you have several options:

1. Raise a feature request on the issues board (we'll address when we can)
2. Fork the repo and build the feature yourself (pull requests are welcomed)
3. Partner with the team at SAS Apps (we build most features in a few hours or days)

The final option of course, is to leave a comment below!

[SASKiwi]

@AllanBowe  - very interesting! I'm curious to know if your IDE extensions would also be compatible with commercial versions of Visual Studio, for example VS Professional 2019? We are using this in preference to Visual Studio Code. 

[AllanBowe]

Hi SASKiwi - glad you like it!  The VS Code extension would not be compatible as VS Pro uses an entirely different SDK.  If there are any readers (with developers, or budget) who would like us to support the VS Pro marketplace, we'd be happy to chat!

That said, you don't need the extension to use the SASjs framework.  All the functionality (minus the syntax highlighting obviously) is also available in the CLI.  You can run `sasjs lint` to fix the code, and `sasjs run` to execute it.  Very (very) soon the SAS execution feature will work for SAS 9, and in a few weeks, for Desktop SAS also.