Python subprocess terminated unexpectedly, trying to use more memory than the container is configure

[asirait]

Hi SAS Community,

We got an error when running Python Job in SAS Studio (SAS Viya 4 LTS 2023.10), which consume large amount of data (there are data query inside the python job). The error is related to the container memory limit:

"ERROR: Python subprocess terminated unexpectedly. Exit code is 265 (0x0109). The most likely cause of this is the Python process trying to use more memory than the container is configured to be allowed to use."

We have tried to increase the memory size of  sas-programming-environment container in sas-compute pod, up to 16GB, increase System options MAXTABLEMEM, MAXMEMQUERY, and MEMSIZE up to 2G but still didn't solve the problem.

Where is the specific container/pod of python process when running python job in SAS Studio, in which we have to enlarge the memory size? Or there are any configuration to solve this problem?

[DerylHollick · Accepted solution]

I hit this same issue once trying to get a Python program migrated to Viya4.  The admin had to increase Launcher Service->sas.launcher.default->memory.limit and Launcher Service->sas.launcher.max->memory.limit from 2GB up to 32GB before this one big geodataframe would load.  I first had to run the program on a Linux machine and see how much memory it used before I could tell the admin how much would be needed to run in the container.

SAS options like MEMSIZE, MAXMEMQUERY, etc. are applicable to how SAS does its processing within the available resources but not for how Python does its processing. 

[asirait]

Thank you for the solution. We have configured the memory of launcher in environment manager as you mentioned, but it is not directly affected the size of the sas-programming-environment memory size in compute pod. Hence, we also increase the memory limit of sas-compute-job-config on launcher.sas.com up to 32Gi by using this script:

kubectl -n scoring-engine annotate PodTemplate sas-compute-job-config --overwrite launcher.sas.com/default-memory-limit=32000Mi

kubectl -n scoring-engine annotate PodTemplate sas-compute-job-config --overwrite launcher.sas.com/max-memory-limit=32000Mi

After these configurations, the python job can be executed successfully.  The actual memory utilization is about 22Gi, so 24Gi for the memory is actually enough.

[AhmedAl_Attar]

@asirait 

To optimize your Python subprocess memory consumptions, you may need to look into these links if your python code uses Pandas.

Using Pandas 2.x with Arrow DataTypes, or using Polars 1.x as an alternative to Pandas Dataframe 

• https://pythonspeed.com/articles/pandas-load-less-data/
• https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i
• https://pub.towardsai.net/pandas-vs-polars-bid-adieu-to-pandas-and-switch-to-polars-6d1f118994fa
• https://medium.com/@husein2709/pandas-2-0-vs-polars-pvp-d8aec54c3fc8
• https://medium.com/@kevinmenesesgonzalez/polars-vs-pandas-the-future-of-data-frames-c508fc4408dc (one more link)

Hope this helps