How to Get the Code From a SAS Studio Flow Using Python

2 Likes

SAS customers need to run a SAS Studio Flow in batch or from a CI/CD pipeline. They also need to version the flow code, possibly in a Git repository and compare the changes from one version to another. To achieve these goals, you need the SAS code behind the SAS Studio Flow. A REST API helps you generate the code. Read the post to find out how you can call the SAS Viya REST API from Python.

The post How To Get the Code From a SAS Studio Flow and Why It Matters explained how to get the SAS code from a SAS Studio Flow by writing a SAS program (proc http). You can write a Python program to achieve the same result.

Python Program Calling SAS Viya REST APIs

The studioDevelopment REST API has been available since the SAS Viya 2022.09 Long Term Support. I tested the program below with Python version 3.8, 3.9 and 3.10.

SAS Studio Flow

Suppose you developed a flow that performs a de-duplication of an existing table.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

To generate the SAS code form this flow you could write a Python program:

The first part of the program returns a SAS Viya Token.
The second part generates the SAS code from a SAS Studio Flow.

Get a SAS Viya Access Token

There are several ways to get a token. I will just pick one method I favor for several reasons.

Packages

Several packages are required for this program to run.

# 1 Packages
import requests
import urllib3
import json, os, pprint
import getpass
import base64

Variables

The unsafe choice is to hardcode credentials in the program. It might be suitable for a quick test, although you should never save it in a program file. The file can end up in a Git repository. The Git repository can become public therefore compromising your credentials.

Never hardcode credentials directly in the program.

Instead of hardcoding variables you can pass them as arguments. Two reasons for using arguments:

Avoid hardcoding.
In production environments, Python programs are executed from the command line most of the time. Therefore, it helps if you pass the “changing” part as an argument.

In this example:

The first argument is the Python program name. More about it in the execution part.
The RG is the second argument ([1] in Python) in the argument array.
The client_id is the third argument ([2] in Python) in the argument array.
The client_secret is the fourth argument ([3] in Python) in the argument array.
The pem_path is the path to the SSL client-side certificate. E.g. '/path_to/my_trustedcerts.pem'.
The FLOW_STR points to the path and name of the flow in SAS Viya. E.g.: Public/Flow.flw.


# 2. Variables
import sys
print ("Number of arguments:", len(sys.argv), "arguments")
#print ("Argument List:", str(sys.argv))
RG=str(sys.argv[1])
client_id=str(sys.argv[2])
client_secret=str(sys.argv[3])
pem_path=str(sys.argv[4])
FLOW_STR=str(sys.argv[5])

# Viya URLs
VIYAURL='https://' + RG +'.sas.com'
url = VIYAURL + "/studioDevelopment/code"
authUri="/SASLogon/oauth/token"

Client and Secret Method

The authentication method in this example uses a SAS Viya registered client, called gel_client3 and a client secret. A SAS Administrator registered the client with "authorized_grant_types": ["client_credentials"].

How to register such a client and the secret? See for more details:

It simply is a more elegant choice to use a client and a client secret to get a SAS Viya access token.

Reasons:

You can use this grant type in the payload sent to the SAS Viya REST API. This is better than using your user and the password. The password grant type is not considered secure and should not be used in production environments.
There are ways to rotate secrets automatically. If the secret is exposed, you can just refresh it. Because you are passing it as a parameter, you do not need to update the program.
You can scope the client registration. The scope links the client to a group and its authorization rules. This acts as a ring-fence around what the client can be used for.


# 3. This is using a client id and a secret - no user and password in the payload
print("The Auth is using This is using a client id and a secret. Grant type: client_credentials")

# client_id must be registered using client_credentials grant type

# base 64 encode the api client id and secret and pass it in the authorization header
auth_str = f'{client_id}:{client_secret}'
auth_bytes = auth_str.encode('ascii')
auth_header = base64.b64encode(auth_bytes).decode('ascii')
my_headers = {'Authorization': f'Basic {auth_header}'}

payload = {
    'grant_type': 'client_credentials',
    'client_id': client_id,
    'client_secret': client_secret
    }

print(f'Logging in to {VIYAURL}')
# post as form
response = requests.post(
    f'{VIYAURL}/SASLogon/oauth/token', data=payload, headers=my_headers, verify=pem_path)
print(response)

token = response.json()['access_token']
#print (token)

In addition, we are using SSL certificate verification in the request: verify=pem_path resolves to verify='/path_to/my_trustedcerts_.pem'. You should specify a local certificate to use as a client-side certificate. The client-side certificate can be obtained using a kubectl command. See Get the SAS Viya Certificates in How to Score a SAS Decision Published to MAS Using Shell Commands.

Generate the SAS Code from a SAS Studio Flow

Now that you have a SAS Viya access token, you can use the /studioDevelopment/code endpoint to extract the SAS code from the SAS Studio Flow.

# 4. Headers containing the token
headers = {
'Authorization': 'bearer ' + token,
'Accept': 'application/json',
'Content-Type': 'application/json;charset=utf-8'
}
#print(headers)

# 5. Payload to be passed with the REST API. The flow is passed as a variable inside.
## FLOW='/path_to/flow_name.flw'. E.g. #FLOW='/Public/Flow.flw'
## We add the '/' to the FLOW_STR inside the program.
### As of 2022.09 the Flow MUST HAVE A TABLE step inside.

FLOW = '/' + FLOW_STR
print(FLOW)

payload_dict = {
    'reference': {
        'type': "content",
        'path': FLOW,
        'mediaType': "application/vnd.sas.dataflow"
    },
    'initCode': False,
    'wrapperCode': False
}

payload = json.dumps(payload_dict)
print(payload)

# 6. POST REQUEST

response = requests.request("POST", url, headers=headers, data=payload, verify=pem_path)

print(response.text)
print(response.json())

# 7. Write the output to a  JSON file

jsonString = json.dumps(response.json())
jsonFile = open("Flow.json", "w")
jsonFile.write(jsonString)
jsonFile.close()

The headers contain the SAS Viya token.

The SAS Studio Flow we want to obtain the code from is stored in /Public/Flow.flw.

A JSON payload is prepared from a dictionary variable payload_dict. The SAS Studio Flow path and name are passed as an argument from the FLOW variable. We then “dump” the dictionary variable in JSON.

{
    "reference": {
        "type": "content",
        "path": "/Public/Flow.flw",
        "mediaType": "application/vnd.sas.dataflow"
    },
    "initCode": false,
    "wrapperCode": false
}

The payload is passed, or posted, to the SAS Viya REST API. The response comes back in JSON format. We write the output to a JSON file.

Execute the Python Program

Suppose you will save the whole Python program as codeGen.py. On a Linux machine, you can call the above script from the command line using the python command, the Python program name, followed by the arguments.

Hardcoded secret

You could simply pass the secret in the command:

python3 codeGen_ssl_args.py 'viya_url_prefix' 'gel_client3' 'the_secret' '/path_to/my_trustedcerts.pem' 'Public/Flow.flw'

Vaulted Secret

If you are calling the script from an Azure Virtual Machine, you should store the secret in an Azure Key Vault. You can then retrieve the secret using the az keyvault secret show command.

secret=$(az keyvault secret show --name THE_SECRET --vault-name THE_KEY_VAULT --query "value" -o tsv)

Then, you can call the Python program using:

python3 codeGen_ssl_args.py 'viya_url_prefix' 'gel_client3' $secret '/path_to/my_trustedcerts.pem' 'Public/Flow.flw'

This approach has the advantage that the secret will not be directly written to an output or captured in a log file.

Clean the JSON Output

The output is written to the Flow.json file. If you display it, it will look a bit like The Matrix code with Latin letters. There are no sushi recipes encoded inside, just SAS code:

cat Flow.json | jq

SAS Studio Flow Code Output in JSON format

Or, you will certainly want the SAS code stripped of separators like \n:

cat Flow.json | jq --raw-output '.code' | sed 's/\\n/\n/g' > Flow.sas
cat Flow.sas

Now, not only it looks better, but you can now run the SAS file in batch, store it to a Git repository and so on.

PyViyaTools

And since we are discussing Python and code generation, Gerry Nelson has just created a new pyviyatool named exportstudioflowcode.py. The nice thing about it is that it can loop through a folder and convert all the SAS Studio Flows from that folder.

# usage: 
exportstudioflowcode.py [-h] -t {Flow,Folder} -n NAME -d DIRECTORY [--includeinitcode] [--includewrappercode]

The pyviyatools are a set of command-line tools that call SAS Viya REST APIs from Python. These tools simplify common administration tasks, provide more complex functionality and are a great complement to the SAS Viya CLI. The tools are available on the public SAS Software GitHub. They are easy to install and use.

Conclusions

The post provided you a Python program you can use to interact with SAS Viya REST APIs.

First, we discussed the importance of avoiding to hardcode credentials in programs, the advantages of clients and secrets and how to pass the secrets as arguments in the Python program.
Second, we obtained a SAS Viya token.
Third, we discussed the SAS Viya codeGen REST API and how to generate the SAS code from a SAS Studio Flow.
Fourth, you executed the Python program using shell and passed the arguments.
Lastly, you cleansed the output and obtained the SAS program from the JSON file.

Acknowledgements

Thank you, Alexey Vodilin and Richard Frazer for several Python code snippets I used as inspiration.

Resources

Find more articles from SAS Global Enablement and Learning here.