topic Re: Data extraction from a particular library on server. in SAS Enterprise Guide

Data extraction from a particular library on server.

Visal_Chourasia — Fri, 03 Nov 2023 03:19:28 GMT

I am trying extract a very large dataset (more than a billion records) from a particular library to my local library. but it is giving me memory and timeout error. I tried to extract it by creating small chunks also but still it is giving me the same error.
I would appreciate if someone provides me any solution to load this data to my local library efficiently.

Thanks

Re: Data extraction from a particular library on server.

AlanC — Fri, 03 Nov 2023 04:08:03 GMT

Since SAS datasets are also ODBC compliant, try pulling them vs pushing them. I am not sure whether SAS would still have an issue but a simple Python, C#, PowerShell apparatus may work. You could also just try a binary copy using robocopy (assuming Windows) and set up timeout parameters. PowerShell is also highly efficient.

At the heart of it, the sas7bdat is just a simple file. Don't try to read and export, but just copy. If you need it in a different form, consider the ODBC route. You don't indicate what your final form should be so hard to tell the needs.

Re: Data extraction from a particular library on server.

Kurt_Bremser — Fri, 03 Nov 2023 09:39:52 GMT

Define "local". Does it mean you want to download the dataset to your desktop PC, or do you want to copy it to a library of yours on the SAS server?

Also post the log from your failed code; include the LIBNAME for your "local" library.

Re: Data extraction from a particular library on server.

LinusH — Fri, 03 Nov 2023 09:55:30 GMT

The fact "more than a billions of records" should trigger you to rethink your use case.

Try to use the data where it resides, and only extract subsets/aggregates.

This can be achieved using SQL pass through (explicit and/or implicit).

Or if the dataset is a SAS dataset, use SAS/CONNECT or other methods to use it directly.