topic Re: Large Dataset in SAS Programming

Large Dataset

Iblouse — Thu, 25 Mar 2021 13:23:38 GMT

Hi, I have a large dataset 1.1 GB (health data from the CDC), I want to load it into SAS Academy OnDemand but the size limit is 1 GB. Is there another method to load it?

Thanks!

Re: Large Dataset

Iblouse — Thu, 25 Mar 2021 13:27:10 GMT

Note: a precision the dataset is a SAS .XPT file.

Thank you!

Re: Large Dataset

Kurt_Bremser — Thu, 25 Mar 2021 15:00:26 GMT

If that file is available for download from a website, consider using PROC HTTP to download it:

filename csvfile "$HOME/class.csv";
proc http
  url="https://communities.sas.com/kntur85557/attachments/kntur85557/programming/226864/1/class.csv"
  out=csvfile
;
run;

(this example uses a file attached to this post)

Re: Large Dataset

ChrisNZ — Sat, 27 Mar 2021 07:38:15 GMT

And consider storing it as a binary-compressed SPDE file.

Re: Large Dataset

Ksharp — Fri, 26 Mar 2021 11:36:45 GMT

1) Try options:
option obs=10000;
proc copy .....

load the first 10000 records. And

option firstobs=10001;
proc copy .........

load the remain records.

2) switch into SAS University Edition .

Re: Large Dataset

japelin — Fri, 26 Mar 2021 15:53:13 GMT

If you don't want to use XPT, but a SAS data set, this program may help.

"XTP2SAS.exe".

https://bitbucket.org/t_kawakami/xpt2sas/src/master/

I made this on the side of binary analysis of SAS XPT files, and it converts XPT files into sas programs that can be run in sas.

The data size of 1.1GB is a concern, but if you upload the sas program to SAS on Demand and run it, it will generate a SAS dataset with the same structure as the XPT file.

screen sampleconverted sas program

Re: Large Dataset

Ksharp — Sat, 27 Mar 2021 10:39:41 GMT

3) Try option:
options compress=yes ;

to make dataset smaller .

Re: Large Dataset

Tom — Sun, 28 Mar 2021 15:28:27 GMT

@Iblouse wrote:

Note: a precision the dataset is a SAS .XPT file.

Thank you!

I assume that your trouble is that you cannot upload the file to your On Demand server because of the size? You could try using compression on the file and upload the compressed file instead. An actual SAS version 5 transport file normally compresses to less than 30% of the original size. You should be able to create either a ZIP file or GZIP file (.gz extension added after the regular extension).

If the file is really a SAS version 5 format XPORT file then you can reference it by pointing a fileref at it using the ZIP engine and then create a libref with the same name using the XPORT engine.

filename cdc zip 'file_from_cdc.xpt.gz' gzip;
libname cdc xport;
proc copy inlib=cdc outlib=work;
run;