Ways to Handle the 100mb Data Upload Restriction in SAS Viya for Learners, Part II
Part II: I stitch. You stitch. We all stich together.
Hello again VFL fans! A reminder of what we’re striving to accomplish in this series. Essentially, we’re finding creative ways to get around the 100mb upload restriction in SAS Viya for Learners (VFL) – our teaching and learning platform for academics. As noted in the first post – found here https://communities.sas.com/t5/SAS-Communities-Library/Ways-to-Handle-the-100mb-Data-Upload-Restrict... – we (the "royal we" as SAS VFL Administrators) have capped the size of individual file uploads in VFL to create speedbumps that ensure that the system doesn’t become clogged with data that no one is utilizing. You’d be surprised how many people upload data to the cloud once – and never look at it again. And, like your PC, when we reach really high-levels of in-memory data, it can significantly slow down compute time for everyone.
All that stated, you have 5G of cloud space in VFL – so we certainly want you to use it! In this section, you’ll see how to stitch together two files. I’ll use files that are less than 100mb – but the example applies to those larger files too.
Picking up from where we left off in Part I, let’s find the two other Zipped files, which are found here: https://github.com/lincolngroves/Data-and-Analytics-for-Good-Journal-v1. To “fit” the VFL upload constraints, I’ve split the Excel file into two pieces and have compressed them. The two file names are now (1) Vulnerable_Populations_DataSet - Part 1.zip and (2) Vulnerable_Populations_DataSet - Part 2.zip.
Upload time! Like last time, find the Explorer icon on the left-pane. And let’s again save the data under casuser – although you can certainly create a new folder and stick the data there. Right click on casuser and choose Upload Data. The clicks are the following:
See my last post if I went through uploading data too quickly. Next select our two new files – and pretend that they’re large… but not too large (i.e., over 100 mb):
Click Upload. Examine your casuser folder, which should have (at least) these three zipped files, if you completed Part I:
Now, let’s use the Chris Hemedinger Blog magic again (see: https://blogs.sas.com/content/sasdummy/2015/05/11/using-filename-zip-to-unzip-and-read-data-files-in...) to unzip the files. I’ll go through this rather fast since we did it in Part I – and just provide the required code. And, of course, I use the file pathway that corresponds to my VFL environment:
%macro hilfe(file1,outfile);
filename inzip ZIP "/shared/home/Lincoln.Groves@sas.com/casuser/&file1..zip";
/* identify a temp folder in the WORK directory */
filename xl "%sysfunc(getoption(work))/&file1..xlsx" ;
/* hat tip: "data _null_" on SAS-L */
data _null_;
/* using member syntax here */
infile inzip(&file1..xlsx)
lrecl=256 recfm=F length=length eof=eof unbuf;
file xl lrecl=256 recfm=N;
input;
put _infile_ $varying256. length;
return;
eof:
stop;
run;
proc import datafile=xl dbms=xlsx out=&outfile replace;
run;
%mend;
%hilfe(Vulnerable_Populations_DataSet - Part 1,Vulnerable_Populations_Part1);
%hilfe(Vulnerable_Populations_DataSet - Part 2,Vulnerable_Populations_Part2);
And, yes, I used some Macros as a sneaky way to expose you to SAS Macros for iterative processing. Sneaky, sneaky – but macros are a wonderful tool too!
Now, let’s ensure that our two new files have been created, by examining the WORK folder:
Red wine, success!
Now it’s time for the stitch – aka combining the individual pieces of data to create your larger file in SAS VFL. Question now is: are you new to SAS Studio? And what do you know about SAS Studio tasks?
Well, SAS Studio tasks are a lifesaver for new coders! Tasks can help you write SAS code, all with a friendly point and click approach. Without further ado, we can find the Tasks icon on the left pane. Expand SAS Tasks as follows:
Look at all those options available! Please explore a bit on your time, and then navigate to Prepare Data >> Transform Data >> Combine Tables:
Double click on Combine Tables to open the SAS Studio Tasks window:
A brief tutorial is in order. On the left pane of the Combine Table tasks, we see a bunch of red, because need to specify a minimum number of input options, such as the first table, the second table, the type of merge, and what type of code we’d like produced. Once that information is provided, SAS will automatically produce the code for us. Yippie!
Note: your SAS Studio view may differ slightly from my screenshots. There are many ways to alter the view in SAS Studio – choose the one that works best for you!
Back to the regular scheduled programming. Follow my lead and choose Vulnerable_Populations_Part1 as the First data table, Vulnerable_Populations_Part2 as the Second data table, and Concatenate as the Select combination method:
Now look to the right – code is created for us!
Success x 2! Moreover, we can click on the Output tab to change the name of the output file. For exposition’s sake, I’ve changed the name of the output Table name to work.VulPop_Combined. Notice that the SAS code is automatically updated:
Go ahead and click the Run code icon to submit the code:
And, voila, we have a new combined data set in the WORK folder!
Now there are many potential learning paths from this point. In my example, two files are combined using a simple “set” statement in SAS. But you may have many files (easily handled by the Set statement – because the column headers are the same and you’re making a “long” file) or need to merge files to create a “wide” file based on select overlapped variables (a bit more complicated). Whatever your adventure, know that we have the tools to support you in your analytical endeavors. Just reach out below – or in another SAS Communities thread.
Good luck!
