Python script crashing in SAS Viya when querying a large dataset

[mk1804]

Hi,

I'm trying to query data in SAS Viya using a python script. I am able to connect to my CAS Library using the swat.CAS() function. When trying to load a large dataset (~10GB) my python program ends up crashing with the following error message:
raise SWATError('Unable to connect to any URL in the list: %s' %
swat.exceptions.SWATError: Unable to connect to any URL in the list:

I've tried querying smaller datasets and i'm able to run my python script with no issues. Is there a way around this, please advise. Thank you! 

[Patrick]

Without really knowing my first assumption is insufficient memory. Not sure if there is a way that can get you more memory for your session. Something to ask your SAS admin. 

And as a test: Run a python script that does nothing else than reading the data into python. If this works then you could investigate where your current script consumes additional memory and if there are options to reduce its hunger.

[mk1804]

Hi Patrick, 
Thank you for your reply. I am able to read data using my script if its a smaller table, i'm running into issues when it comes to larger tables. I'll reach out to my SAS admin regarding the memory, thanks! 

[AhmedAl_Attar]

Hi @mk1804 

One thing to keep in mind is Viya SWAT package functions would return/send Pandas "Dataframe" object, well it's SASDataFrame which is a subclass of pandas.DataFrame to your Python process.

Without knowing which version of the SWAT package you are using, the SWAT docs indicates pandas.DataFrame v 0.192.2 being used! 

Therefore, I would highly recommend you look into techniques/ways to reduce your pandas DataFrame memory footprint. 

https://www.sparkcodehub.com/pandas/advanced/memory-usage-guide

Here are additional resources that might be of use to you and other readers
Getting Started with Python Integration to SAS® Viya® - Part 2 - Working with CAS Actions and CASRes...
Getting Started with Python Integration to SAS® Viya® - Part 14 - CAS Table to DataFrame 

Hope this helps,

Ahmed

[mk1804]

Thank you Ahmed - will take a look at the links!