Large Dataset

[Iblouse]

Hi, I have a large dataset 1.1 GB (health data from the CDC), I want to load it into SAS Academy OnDemand but the size limit is 1 GB. Is there another method to load it?

Thanks!  

[Iblouse]

Note: a precision the dataset is a SAS .XPT file.

Thank you!

[Tom]

---

@Iblouse wrote:

Note: a precision the dataset is a SAS .XPT file.

Thank you!

---

I assume that your trouble is that you cannot upload the file to your On Demand server because of the size?  You could try using compression on the file and upload the compressed file instead. An actual SAS version 5 transport file normally compresses to less than 30% of the original size. You should be able to create either a ZIP file or GZIP file (.gz extension added after the regular extension).

If the file is really a SAS version 5 format XPORT file then you can reference it by pointing a fileref at it using the ZIP engine and then create a libref with the same name using the XPORT engine.

filename cdc zip 'file_from_cdc.xpt.gz' gzip;
libname cdc xport;
proc copy inlib=cdc outlib=work;
run;

[Kurt_Bremser]

If that file is available for download from a website, consider using PROC HTTP to download it:

filename csvfile "$HOME/class.csv";
proc http
  url="https://communities.sas.com/kntur85557/attachments/kntur85557/programming/226864/1/class.csv"
  out=csvfile
;
run;

(this example uses a file attached to this post)

[ChrisNZ]

And consider storing it as a binary-compressed SPDE file.

[Ksharp]

1) Try options:
option obs=10000;
proc copy .....

load the first 10000 records. And

option firstobs=10001;
proc copy .........

load the remain records.

2) switch into SAS University Edition .

[japelin]

If you don't want to use XPT, but a SAS data set, this program may help.

"XTP2SAS.exe".

https://bitbucket.org/t_kawakami/xpt2sas/src/master/

I made this on the side of binary analysis of SAS XPT files, and it converts XPT files into sas programs that can be run in sas.

The data size of 1.1GB is a concern, but if you upload the sas program to SAS on Demand and run it, it will generate a SAS dataset with the same structure as the XPT file.

screen sampleconverted sas program

[Ksharp]

3) Try option:
options compress=yes ;

to make dataset smaller .