Hi SAS Community,
We got an error when running Python Job in SAS Studio (SAS Viya 4 LTS 2023.10), which consume large amount of data (there are data query inside the python job). The error is related to the container memory limit:
"ERROR: Python subprocess terminated unexpectedly. Exit code is 265 (0x0109). The most likely cause of this is the Python process trying to use more memory than the container is configured to be allowed to use."
We have tried to increase the memory size of sas-programming-environment container in sas-compute pod, up to 16GB, increase System options MAXTABLEMEM, MAXMEMQUERY, and MEMSIZE up to 2G but still didn't solve the problem.
Where is the specific container/pod of python process when running python job in SAS Studio, in which we have to enlarge the memory size? Or there are any configuration to solve this problem?
I hit this same issue once trying to get a Python program migrated to Viya4. The admin had to increase Launcher Service->sas.launcher.default->memory.limit and Launcher Service->sas.launcher.max->memory.limit from 2GB up to 32GB before this one big geodataframe would load. I first had to run the program on a Linux machine and see how much memory it used before I could tell the admin how much would be needed to run in the container.
SAS options like MEMSIZE, MAXMEMQUERY, etc. are applicable to how SAS does its processing within the available resources but not for how Python does its processing.
I hit this same issue once trying to get a Python program migrated to Viya4. The admin had to increase Launcher Service->sas.launcher.default->memory.limit and Launcher Service->sas.launcher.max->memory.limit from 2GB up to 32GB before this one big geodataframe would load. I first had to run the program on a Linux machine and see how much memory it used before I could tell the admin how much would be needed to run in the container.
SAS options like MEMSIZE, MAXMEMQUERY, etc. are applicable to how SAS does its processing within the available resources but not for how Python does its processing.
Thank you for the solution. We have configured the memory of launcher in environment manager as you mentioned, but it is not directly affected the size of the sas-programming-environment memory size in compute pod. Hence, we also increase the memory limit of sas-compute-job-config on launcher.sas.com up to 32Gi by using this script:
kubectl -n scoring-engine annotate PodTemplate sas-compute-job-config --overwrite launcher.sas.com/default-memory-limit=32000Mi
kubectl -n scoring-engine annotate PodTemplate sas-compute-job-config --overwrite launcher.sas.com/max-memory-limit=32000Mi
After these configurations, the python job can be executed successfully. The actual memory utilization is about 22Gi, so 24Gi for the memory is actually enough.
To optimize your Python subprocess memory consumptions, you may need to look into these links if your python code uses Pandas.
Using Pandas 2.x with Arrow DataTypes, or using Polars 1.x as an alternative to Pandas Dataframe
Hope this helps
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.