topic Python script crashing in SAS Viya when querying a large dataset in SAS Viya

Python script crashing in SAS Viya when querying a large dataset

mk1804 — Wed, 18 Jun 2025 21:27:54 GMT

Hi,

I'm trying to query data in SAS Viya using a python script. I am able to connect to my CAS Library using the swat.CAS() function. When trying to load a large dataset (~10GB) my python program ends up crashing with the following error message:
raise SWATError('Unable to connect to any URL in the list: %s' %
swat.exceptions.SWATError: Unable to connect to any URL in the list:

I've tried querying smaller datasets and i'm able to run my python script with no issues. Is there a way around this, please advise. Thank you!

Re: Python script crashing in SAS Viya when querying a large dataset

Patrick — Thu, 19 Jun 2025 11:36:24 GMT

Without really knowing my first assumption is insufficient memory. Not sure if there is a way that can get you more memory for your session. Something to ask your SAS admin.

And as a test: Run a python script that does nothing else than reading the data into python. If this works then you could investigate where your current script consumes additional memory and if there are options to reduce its hunger.

Re: Python script crashing in SAS Viya when querying a large dataset

AhmedAl_Attar — Fri, 20 Jun 2025 13:12:10 GMT

Hi @mk1804

One thing to keep in mind is Viya SWAT package functions would return/send Pandas "Dataframe" object, well it's SASDataFrame which is a subclass of pandas.DataFrame to your Python process.

Without knowing which version of the SWAT package you are using, the SWAT docs indicates pandas.DataFrame v 0.192.2 being used!

Therefore, I would highly recommend you look into techniques/ways to reduce your pandas DataFrame memory footprint.

https://www.sparkcodehub.com/pandas/advanced/memory-usage-guide

Here are additional resources that might be of use to you and other readers
Getting Started with Python Integration to SAS® Viya® - Part 2 - Working with CAS Actions and CASResults Objects
Getting Started with Python Integration to SAS® Viya® - Part 14 - CAS Table to DataFrame

Hope this helps,

Ahmed

Re: Python script crashing in SAS Viya when querying a large dataset

mk1804 — Wed, 25 Jun 2025 00:52:58 GMT

Hi Patrick,
Thank you for your reply. I am able to read data using my script if its a smaller table, i'm running into issues when it comes to larger tables. I'll reach out to my SAS admin regarding the memory, thanks!

Re: Python script crashing in SAS Viya when querying a large dataset

mk1804 — Wed, 25 Jun 2025 00:54:27 GMT

Thank you Ahmed - will take a look at the links!