BookmarkSubscribeRSS Feed

Submit Python code to SAS Viya from the command line

Started ‎04-11-2023 by
Modified ‎10-21-2024 by
Views 2,772

Update to post: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.

 

One of the key features of SAS Viya is its ability to integrate with different programming languages, including Python. SAS users can submit Python code for execution in SAS Viya with the PYTHON procedure; check out this blog post for an overview of what's possible. Some configuration changes are required to get set up, after which users can start running Python code from a SAS Compute session. That's great for submitting from SAS Studio, but what about in batch?

 

The necessary setup is covered nicely by Scott McCauley in his article. When the steps are completed, users can submit Python code or programs for execution in a SAS Compute pod (e.g. in a SAS Studio session).

 

However, a peek inside the file site-config/sas-open-source-config/python/python-transformer.yaml suggests that these configuration changes will only apply to the "sas-compute-job-config" PodTemplate; that's the Custom Resources that tells Kubernetes how to initialise a SAS Compute pod. Any new SAS Compute pod that gets launched will contain these changes to support Python code execution. However, things work a little differently with batch submission of code. The sas-viya CLI has a batch plugin that allows SAS programs (which can contain a PROC PYTHON statement to embed Python code) to be submitted from the command line. With this method, the submitting user must specify a batch context (as opposed to a compute context) to use. By default, Viya is deployed with two batch contexts; default and default-cmd. Both of these use the "sas-batch-pod-template" PodTemplate to initialise a batch session to execute the request. Unfortunately, this template will not contain the changes made to the compute template that are necessary to run Python or R.

 

We can see this difference by comparing the the compute PodTemplate with the batch PodTemplate.

 

Running kubectl describe podtemplate sas-compute-job-config shows all the Python configuration, such as the required volume mounts, is present:

 

AF_1_computepodtemplate_pyconfig-1024x841.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

 

Those are missing in the output shown by running kubectl describe podtemplate sas-batch-pod-template. This means that when we try to run a SAS program containing Python code, it will fail, since none of the required configuration will exist in the pod launched to run the code.

 

The solution, then, is to modify the batch PodTemplate so that batch sessions also contain the required open source configuration.  We can just copy and paste the lines that already exist in the transformer for patching the compute PodTemplate to also patch the batch PodTemplate.

 

The lines of interest are towards the end of python-transformer.yaml:

 

 

 

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: compute-job-python-transformer
patch: |-
  # Add python volume
  - op: add
    path: /template/spec/volumes/-
    value:
      name: python-volume
      persistentVolumeClaim:
        claimName: sas-pyconfig

  # Add mount path for python
  - op: add
    path: /template/spec/containers/0/volumeMounts/-
    value:
      name: python-volume
      mountPath: /opt/sas/viya/home/sas-pyconfig
      readOnly: true

  # Add python-config configMap
  - op: add
    path: /template/spec/containers/0/envFrom/-
    value:
      configMapRef:
        name: sas-open-source-config-python

target:
  kind: PodTemplate
  name: sas-compute-job-config
  version: v1

 

 

 

After copy and pasting the above section, and then editing to target the batch PodTemplate, the end of your python-transformer.yaml should now also include:

 

 

 

...
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: enable-python-batch-transformer
patch: |-
  # Add python volume
  - op: add
    path: /template/spec/volumes/-
    value:
      name: python-volume
      persistentVolumeClaim:
        claimName: sas-pyconfig

  # Add mount path for python
  - op: add
    path: /template/spec/containers/0/volumeMounts/-
    value:
      name: python-volume
      mountPath: /opt/sas/viya/home/sas-pyconfig
      readOnly: true

  # Add python-config configMap
  - op: add
    path: /template/spec/containers/0/envFrom/-
    value:
      configMapRef:
        name: sas-open-source-config-python

target:
  kind: PodTemplate
  name: sas-batch-pod-template
  version: v1

 

 

 

If you've done all the earlier pre-req steps, your kustomization.yaml file won't require any further changes, so you can just build and apply (or deploy your sasdeployment custom resource if you have the SAS Deployment Operator configured).

 

Now let's verify the changes. When deployment completes, run kubectl describe podtemplate sas-batch-pod-template|grep -A5 -B5 sas-pyconfig to check that the batch PodTemplate has been updated:

 

 

 

   k:{"mountPath":"/opt/sas/viya/home/commonfiles"}:
                  .:
                  f:mountPath:
                  f:name:
                  f:readOnly:
                k:{"mountPath":"/opt/sas/viya/home/sas-pyconfig"}:
                  .:
                  f:mountPath:
                  f:name:
                  f:readOnly:
                k:{"mountPath":"/sashelp"}:
--
        Mount Path:  /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private
        Name:        security
        Sub Path:    private
        Mount Path:  /gelcontent
        Name:        sas-viya-gelcorp-volume
        Mount Path:  /opt/sas/viya/home/sas-pyconfig
        Name:        python-volume
        Read Only:   true
    Dns Policy:      ClusterFirst
    Image Pull Secrets:
      Name:  sas-image-pull-secrets-k95ckd46h8
--
      Nfs:
        Path:    /shared/gelcontent
        Server:  pdcesx02072.race.sas.com
      Name:      python-volume
      Persistent Volume Claim:
        Claim Name:  sas-pyconfig
Events:              

 

 

 

Much better. Then the final test; checking to see if batch submission works.

 

The program we want to run looks like:

 

 

 

proc python;
submit;
print('hello world')
endsubmit;
run;

 

 

 

The command that runs it looks like:


sas-viya batch jobs submit-pgm --pgm-path /shared/gelcontent/gelcorp/shared/code/my_code.sas --context default --watch-output --wait-log-list --results-dir ~/

 

And that results in:

 

 

 

>>> The file set "JOB_20230406_102344_048_1" was created.
>>>   Uploading "test.sas".
>>> The job "1dda5231-1978-4ab0-9da4-d0e264fdecde" was submitted.
>>> Waiting for the job to complete...
>>> Job started.
1                                                          The SAS System                       Thursday, April  6, 2023 10:23:00 AM

NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.04.00 (TS M0 MBCS3170)
      Licensed to THIS ORDER IS FOR SAS INTERNAL USE ONLY (SIMPLE), Site 00000000.
NOTE: This session is executing on the Linux 3.10.0-1062.12.1.el7.x86_64 (LIN X64) platform.



NOTE: Additional host information:

 Linux LIN X64 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 Red Hat Enterprise Linux release 8.7 (Ootpa)

NOTE: SAS initialization used:
      real time           0.02 seconds
      cpu time            0.02 seconds


NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/sasgrid/default/autoexec.sas.


NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds




NOTE: AUTOEXEC processing completed.

NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state.

NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state.

1          proc python;
2          submit
NOTE: Python initialized.
Python 3.8.13 (default, Apr  4 2023, 04:53:36)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
2        !       ;
3          print('hello world')
4          endsubmit;
5          run;

6          %let SAS_workpath = %sysfunc(pathname(work));

>>>
hello world
>>>
NOTE: PROCEDURE PYTHON used (Total process time):
      real time           1.36 seconds
      cpu time            0.01 seconds


NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
      real time           1.40 seconds
      cpu time            0.05 seconds
2                                                          The SAS System                       Thursday, April  6, 2023 10:23:00 AM


<<< Job ended.
<<<   Downloading the files in the file set to the following path: "/home/cloud-user//JOB_20230406_102344_048_1".
<<<   Downloading "test.log".
<<<   Downloading "test.sas".
<<<   Copying the log and list files from "/home/cloud-user//JOB_20230406_102344_048_1" to the current directory.
<<<   Copying "test.log".
<<< Deleting the results directory.
<<< Deleting the batch job.
<<< Deleting the file set.

 

 

 

Success!

 

There's probably a solid enough case for having the open source configuration applied to both the compute and batch PodTemplates out-of-the-box with the python-transformer.yaml; which will obviously simplify and reduce the setup tasks required. I will update this post with the revised steps if the default setup changes. 

Update: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.

 

Thanks for reading. My thanks also to my colleaque @Bogdan_Teleuca for his contributions.

 

Find more articles from SAS Global Enablement and Learning here.

Version history
Last update:
‎10-21-2024 04:38 AM
Updated by:
Contributors

SAS Innovate 2025: Register Now

Registration is now open for SAS Innovate 2025 , our biggest and most exciting global event of the year! Join us in Orlando, FL, May 6-9.
Sign up by Dec. 31 to get the 2024 rate of just $495.
Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags