Uploading large files into SAS Studio ODA

[pangeabird]

Hello,

I am trying to import BRFSS 2020 dataset to SAS Studio OnDemand for Academics. A few past posts have commented that they were not able to upload the large files (whether ASCII for .XPT) to SAS Studio University Edition. The BRFSS 2020 dataset (ASCII file) I am trying to upload to SAS is 800 MB. It does not seem to upload to SAS. After more than 30-50 minutes I get timed out. Same when I try to upload the .XPT transport version that I have downloaded and unzipped from CDC BRFSS website. I thought that I have 5GB limit for SAS Studio ODA as a student. Or is the file limit still only 10MB as some have pointed out in this forum in the past? If it is only 10MB, then how would users upload large files? Thank you in advance for your help. 

[Tom]

If you are having upload issues then upload a ZIP file as that will be much smaller. Both text files and XPT files will compress to between 10 and 20 percent of the original file size.  You can use the ZIP filename engine in SAS to access the content of the ZIP file.  For the XPT file you might need to first copy it to an physical file in your WORK directory to be able to access with the XPORT libname engine.

If the file is available via a URL then try running a SAS step to copy the file from the URL to the SAS ODA server and just skip the step of downloading it to you personal computer.  Either use the URL filename engine or use PROC HTTP.

So for example if you have named brfsss.txt then compress it into a ZIP file.  Say you called that file brfsss.zip.  Then to read the file from the zip file your SAS data step could look like:

data want;
  infile '~/brfsss.zip' zip member='brfsss.txt' ;
  input .... ;
run;

If you have a file named brfsss.xpt then you might need to first copy it out of the ZIP file to use it.

filename xpt temp;
data _null_;
  infile '~/brfsss.zip' zip member='brfsss.xpt' lrecl=80 recfm=f ;
  file xpt lrecl=80 recfm=f;
  input;
  put _infile_;
run;
libname xpt xport "%sysfunc(pathname(xpt))";
proc copy inlib=xpt outlib=work;
run;

[pangeabird]

Thank you!

[sbxkoenk · Accepted solution]

Hello @pangeabird ,

If you are working in OnDemand for Academics (ODA) you cannot change the configuration (to increase the maxUploadSize parameter).

But for these people that can change the default limit, I wanted to share the relevant piece of doc that tells you how to do this:
SAS® Viya® 3.5 Administration | 3.5
Configuration Properties
SAS Studio 5.2 (Enterprise)
https://go.documentation.sas.com/doc/en/calcdc/3.5/calconfig/n08025sasconfiguration0admin.htm#n08569...

To set a value for the maximum size of a file that can be uploaded, do the following:
1. Sign in to SAS Environment Manager as an administrative user
2. Go to [ Configuration ] > [ All services ] > [SAS Studio Viya]
3. Type "sas.studiov" in the search box in the right pane
4. Click [ Edit ] icon for the "sas.studiov" configuration
5. Specify a value for the maxUploadSize property.
6. Click [ Save ] to save the configuration

Thanks,
Koen

[ythum]

Not sure who you mean by "these people". 

Are SAS ODA users stuck with upload file sizes at 1G? If so, is SAS Studio Enterprise available by subscription?

Thanks.

[sbxkoenk]

"these people" are SAS administrators.
Only SAS administrators have the full-blown SAS Environment Manager and can change all system options.

SAS® Viya® 3.5 Administration
https://go.documentation.sas.com/doc/en/calcdc/3.5/calwlcm/home.htm

Koen

[ballardw]

The academic part of the name indicates it is for LEARNING, not production job work. Local installs of licensed SAS studio are controlled by the local administrators for such things and wouldn't have the file size limits unless the administrators wanted them.

The 800Mbyt BRFSS data would indicate an entire nationwide data set and such would not be typically needed for a LEARNING environment when anything needed for teaching would work with one or two state data sets.

[ythum]

Thanks for the response. 

Not sure what the BRFSS is or why it is relevant. As for relating file size to "learning", your point is well taken, but large files do not always indicate "production work," or do they indicative "nationwide" numbers. Such wild speculations (at best) are not necessary in a response about program limits.

Exploring the limits of SAS ODA is my learning objective.

[Tom]

---

@ythum wrote:

Thanks for the response. 

Not sure what the BRFSS is or why it is relevant. As for relating file size to "learning", your point is well taken, but large files do not always indicate "production work," or do they indicative "nationwide" numbers. Such wild speculations (at best) are not necessary in a response about program limits.

Exploring the limits of SAS ODA is my learning objective.

---

BRFSS is what this thread from 2021 is about.

If you have a new question then start a new thread.

[ythum]

Just trying to get an answer, to the topic of discussion "RE: Uploading large files into SAS Studio ODA." Forgive me for not going back to the original post IN 2021 for all of that!

My point about large file size ^= nationwide numbers is not unreasonable.