Submit Python code to SAS Viya from the command line
Update to post: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.
One of the key features of SAS Viya is its ability to integrate with different programming languages, including Python. SAS users can submit Python code for execution in SAS Viya with the PYTHON procedure; check out this blog post for an overview of what's possible. Some configuration changes are required to get set up, after which users can start running Python code from a SAS Compute session. That's great for submitting from SAS Studio, but what about in batch?
The necessary setup is covered nicely by Scott McCauley in his article. When the steps are completed, users can submit Python code or programs for execution in a SAS Compute pod (e.g. in a SAS Studio session).
However, a peek inside the file site-config/sas-open-source-config/python/python-transformer.yaml suggests that these configuration changes will only apply to the "sas-compute-job-config" PodTemplate; that's the Custom Resources that tells Kubernetes how to initialise a SAS Compute pod. Any new SAS Compute pod that gets launched will contain these changes to support Python code execution. However, things work a little differently with batch submission of code. The sas-viya CLI has a batch plugin that allows SAS programs (which can contain a PROC PYTHON statement to embed Python code) to be submitted from the command line. With this method, the submitting user must specify a batch context (as opposed to a compute context) to use. By default, Viya is deployed with two batch contexts; default and default-cmd. Both of these use the "sas-batch-pod-template" PodTemplate to initialise a batch session to execute the request. Unfortunately, this template will not contain the changes made to the compute template that are necessary to run Python or R.
We can see this difference by comparing the the compute PodTemplate with the batch PodTemplate.
Running kubectl describe podtemplate sas-compute-job-config shows all the Python configuration, such as the required volume mounts, is present:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Those are missing in the output shown by running kubectl describe podtemplate sas-batch-pod-template. This means that when we try to run a SAS program containing Python code, it will fail, since none of the required configuration will exist in the pod launched to run the code.
The solution, then, is to modify the batch PodTemplate so that batch sessions also contain the required open source configuration. We can just copy and paste the lines that already exist in the transformer for patching the compute PodTemplate to also patch the batch PodTemplate.
The lines of interest are towards the end of python-transformer.yaml:
apiVersion: builtin
kind: PatchTransformer
metadata:
name: compute-job-python-transformer
patch: |-
# Add python volume
- op: add
path: /template/spec/volumes/-
value:
name: python-volume
persistentVolumeClaim:
claimName: sas-pyconfig
# Add mount path for python
- op: add
path: /template/spec/containers/0/volumeMounts/-
value:
name: python-volume
mountPath: /opt/sas/viya/home/sas-pyconfig
readOnly: true
# Add python-config configMap
- op: add
path: /template/spec/containers/0/envFrom/-
value:
configMapRef:
name: sas-open-source-config-python
target:
kind: PodTemplate
name: sas-compute-job-config
version: v1
After copy and pasting the above section, and then editing to target the batch PodTemplate, the end of your python-transformer.yaml should now also include:
...
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: enable-python-batch-transformer
patch: |-
# Add python volume
- op: add
path: /template/spec/volumes/-
value:
name: python-volume
persistentVolumeClaim:
claimName: sas-pyconfig
# Add mount path for python
- op: add
path: /template/spec/containers/0/volumeMounts/-
value:
name: python-volume
mountPath: /opt/sas/viya/home/sas-pyconfig
readOnly: true
# Add python-config configMap
- op: add
path: /template/spec/containers/0/envFrom/-
value:
configMapRef:
name: sas-open-source-config-python
target:
kind: PodTemplate
name: sas-batch-pod-template
version: v1
If you've done all the earlier pre-req steps, your kustomization.yaml file won't require any further changes, so you can just build and apply (or deploy your sasdeployment custom resource if you have the SAS Deployment Operator configured).
Now let's verify the changes. When deployment completes, run kubectl describe podtemplate sas-batch-pod-template|grep -A5 -B5 sas-pyconfig to check that the batch PodTemplate has been updated:
k:{"mountPath":"/opt/sas/viya/home/commonfiles"}:
.:
f:mountPath:
f:name:
f:readOnly:
k:{"mountPath":"/opt/sas/viya/home/sas-pyconfig"}:
.:
f:mountPath:
f:name:
f:readOnly:
k:{"mountPath":"/sashelp"}:
--
Mount Path: /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private
Name: security
Sub Path: private
Mount Path: /gelcontent
Name: sas-viya-gelcorp-volume
Mount Path: /opt/sas/viya/home/sas-pyconfig
Name: python-volume
Read Only: true
Dns Policy: ClusterFirst
Image Pull Secrets:
Name: sas-image-pull-secrets-k95ckd46h8
--
Nfs:
Path: /shared/gelcontent
Server: pdcesx02072.race.sas.com
Name: python-volume
Persistent Volume Claim:
Claim Name: sas-pyconfig
Events:
Much better. Then the final test; checking to see if batch submission works.
The program we want to run looks like:
proc python;
submit;
print('hello world')
endsubmit;
run;
The command that runs it looks like:
sas-viya batch jobs submit-pgm --pgm-path /shared/gelcontent/gelcorp/shared/code/my_code.sas --context default --watch-output --wait-log-list --results-dir ~/
And that results in:
>>> The file set "JOB_20230406_102344_048_1" was created.
>>> Uploading "test.sas".
>>> The job "1dda5231-1978-4ab0-9da4-d0e264fdecde" was submitted.
>>> Waiting for the job to complete...
>>> Job started.
1 The SAS System Thursday, April 6, 2023 10:23:00 AM
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.04.00 (TS M0 MBCS3170)
Licensed to THIS ORDER IS FOR SAS INTERNAL USE ONLY (SIMPLE), Site 00000000.
NOTE: This session is executing on the Linux 3.10.0-1062.12.1.el7.x86_64 (LIN X64) platform.
NOTE: Additional host information:
Linux LIN X64 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 Red Hat Enterprise Linux release 8.7 (Ootpa)
NOTE: SAS initialization used:
real time 0.02 seconds
cpu time 0.02 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/sasgrid/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
NOTE: AUTOEXEC processing completed.
NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state.
NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state.
1 proc python;
2 submit
NOTE: Python initialized.
Python 3.8.13 (default, Apr 4 2023, 04:53:36)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
2 ! ;
3 print('hello world')
4 endsubmit;
5 run;
6 %let SAS_workpath = %sysfunc(pathname(work));
>>>
hello world
>>>
NOTE: PROCEDURE PYTHON used (Total process time):
real time 1.36 seconds
cpu time 0.01 seconds
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 1.40 seconds
cpu time 0.05 seconds
2 The SAS System Thursday, April 6, 2023 10:23:00 AM
<<< Job ended.
<<< Downloading the files in the file set to the following path: "/home/cloud-user//JOB_20230406_102344_048_1".
<<< Downloading "test.log".
<<< Downloading "test.sas".
<<< Copying the log and list files from "/home/cloud-user//JOB_20230406_102344_048_1" to the current directory.
<<< Copying "test.log".
<<< Deleting the results directory.
<<< Deleting the batch job.
<<< Deleting the file set.
Success!
There's probably a solid enough case for having the open source configuration applied to both the compute and batch PodTemplates out-of-the-box with the python-transformer.yaml; which will obviously simplify and reduce the setup tasks required. I will update this post with the revised steps if the default setup changes.
Update: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.
Thanks for reading. My thanks also to my colleaque @Bogdan_Teleuca for his contributions.
Find more articles from SAS Global Enablement and Learning here.
