Bulk Download all my files from SAS Ondemand

[LauraRK]

I taught my class last semester using SAS Ondemand and now I want to download all the files we used to archive them.  Please tell me I do not have to click on each file one at a time.  I have spent time looking but have yet to find away to download all my files from SAS Ondemand to my local machine without clicking each one.  My question is, how can one download all ones files from SAS Ondemand for Academics without having to download them one at a time?  I am a professor and I was using SAS studio.

[RandyMullis]

Hi, @LauraRK . I hate to be the bearer of bad news. There isn't a facility to do this right now. I'm sorry. 

Randy

[LauraRK]

Thank you Randy.

I asked in the programming section if there is away to zip directories and keep directory structures.  Given I am teaching I set up examples and exercises in lots of folders.  I wish I had realized this earlier as I would never have used SAS OnDemand to set up the class.  I would have just kept everything local.  This is a pretty big workflow challenge.

[RandyMullis]

I agree, Laura, it's a big need. The operations team tells me it's on their futures list. I think it would be a great challenge for a good SAS programmer. I've seen several programs that do this, but not in the kind of robust way that would be generally usable. I'm sorry this is an obstacle for you. Best to you in your upcoming teaching.

Randy

[LauraRK]

I think other potential ways would be github integration or integrate with dropbox, or someway a person can save files and have a project workflow and then download that work. One of the biggest challenges in teaching and programming is workflow.
Minimally, I think SAS should have a warning that one should not plan on easily downloading their files in the set up documents for SAS ondemand.

[Kurt_Bremser]

What kind of files to you need to add to the ZIP archive? If it's only SAS programs (text), there might be a not-so-complicated way.

[LauraRK]

If I can minimally do the SAS files and ideally also the directory structure that would be great. I know I have spent more time on this than I would have spent just doing this by hand one at a time, but if I am going to keep using the platform I need to figure out a workflow. I also would love to be able to provide this tool to my students, so they can zip their work after they are done with the class. Thank you so much.

[Kurt_Bremser · Accepted solution]

I just cobbled together a method to create an archive that contains all program files in a directory tree.

First, I used a program created by @Tom that is a very ingenious way of recursively scanning a directory tree in one data step:

data filelist;
  length dname filename $256 dir level 8 lastmod size 8;
  format lastmod datetime20.;
  input dname;
  retain filename ' ' level 0 dir 1;
cards4;
/folders/myfolders
;;;;

data filelist;
  modify filelist;
  rc1=filename('tmp',catx('/',dname,filename));
  rc2=dopen('tmp');
  dir = not not rc2;
  if not dir then do;
    fid=fopen('tmp','i',0,'b');
    lastmod=input(finfo(fid,foptname(fid,5)),NLDATM100.);
    size=input(finfo(fid,foptname(fid,6)),32.);
    fid=fclose(fid);
  end;
  else do;
    dname=catx('/',dname,filename);
    filename=' ';
    lastmod=input(dinfo(rc2,doptname(rc2,5)),NLDATM100.);
  end;
  replace;
  if dir;
  level=level+1;
  do i=1 to dnum(rc2);
    filename=dread(rc2,i);
    output;
  end;
  rc3=dclose(rc2);
run;

Then, I used the resulting dataset to run a macro repeatedly that adds members to a ZIP archive:

filename out zip "/folders/myfolders/all.zip";

%macro write_single(fname);

data _null_;
infile "&fname";
file out("&fname");
input;
put _infile_;
run;

%mend;

data _null_;
set filelist;
where filename ne "" and scan(filename,-1,".") = "sas";
call execute('%write_single('!!catx("/",dname,filename)!!")");
run;

[LauraRK]

For some reason I could not get it to work on SAS ondemand?

It runs and I get no errors but then my zip file is empty.  I do get the file list.  I do appreciate so much your help.  I have some meetings for the rest of the day but I will work on this more later tonight or tomorrow, and try it locally as well. 

[Kurt_Bremser]

I just did a test run on my On Demand account; I only replaced the base paths (/folders/myfolders) with the path to my On Demand home directory, and successfully created a zip file with the .sas program files that I have there.

Everything worked fine, I could extract the two .sas files after downloading to my laptop.

Please take a close look at your log; for each .sas file found, you should have this from the ZIP step:

 NOTE: CALL EXECUTE generated line.
 1         + data _null_; infile "/home/XXX/programs/zip_all.sas"; file out("/home/XXX/programs/zip_all.sas"); 
 input; put _infile_; run;
 
 NOTE: The infile "/home/XXX/programs/zip_all.sas" is:
       Dateiname=/home/XXX/programs/zip_all.sas,
       Besitzername=XXX,Gruppenname=oda,
       Zugriffsberechtigung=-rw-r--r--,
       Zuletzt geändert=26. Jänner 2021 09.39 Uhr,
       Dateigröße (Byte)=312
 
 NOTE: The file library OUT is:
       Directory=/home/XXX/all.zip
 
 NOTE: The file OUT("/home/XXX/programs/zip_all.sas") is:
       Filename=/home/XXX/all.zip,
       Member Name=/home/XXX/programs/zip_all.sas
 
 NOTE: 18 records were read from the infile "/home/XXX/programs/zip_all.sas".
       The minimum record length was 0.
       The maximum record length was 62.
 NOTE: A total of 18 records were written to the file library OUT.
       The minimum record length was 0.
       The maximum record length was 62.
 NOTE: 18 records were written to the file OUT("/home/XXX/programs/zip_all.sas").
       The minimum record length was 0.
       The maximum record length was 62.

(home path redacted)

[LauraRK]

Thank you so much. I got it to work once I moved the files out of my shared folder and into my home directory.  I had the correct paths but I think with the shared drive there is some special protection because I was getting a very odd error. 

This is very helpful and I appreciate your work on this very much.

[Kurt_Bremser]

Could you show the "very odd error"? This might enable us to point to a solution that does not require moving all the files.

[LauraRK]

I should have copied it at the time.  When I restarted SAS Ondemand it no longer was there, but I also no longer had write access to my shared drive.  I think that perhaps what happens is that since my class ended maybe my permissions to my drive changed in the last couple days? 

What I did figure out is that it will work with the shared drive as long as I use the ~ and do not use the copied path. 

If I use the below:

data filelist;
  length dname filename $256 dir level 8 lastmod size 8;
  format lastmod datetime20.;
  input dname;
  retain filename ' ' level 0 dir 1;
cards4;
 /home/u49xxxxxx/my_shared_file_links/kxxxxxx/sta216
;;;;

my zip folder is empty.  Although, it does create a listing of the files in the folders they do not end up copied into the zip file.

However, if I do:

data filelist;
  length dname filename $256 dir level 8 lastmod size 8;
  format lastmod datetime20.;
  input dname;
  retain filename ' ' level 0 dir 1;
cards4;
~/my_shared_file_links/kxxxxxx/sta216
;;;;

It completely works.  So your solution works great.  Next time I will make sure I do this while I still have proper access to my drives.  This was my first semester using SAS OnDemand, I am an experienced SAS user but there was a bit of a learning curve with using drives that I could not access directly outside of SAS.  I very much appreciate this exercise, and your help on this Kurt.  This is nice because I can give this tool to my students to run as well.  

My next question is can I do this in reverse?  It seems I can, but I would need to have sas create the directories as I unzipped.  Perhaps using a libname and dlcreatedir option?  I also would love to be able to upload zipped data as well, not just SAS programs and unzip it on SAS Ondemand but I do not see that option yet that I can find.

Thanks so much for your help.  I am very appreciative.

[Kurt_Bremser]

Maybe they did something weird with the contents of $HOME. The tilde returns the contents of that environment variable.

[RandyMullis]

Hi, @Kurt_Bremser and @LauraRK . The permissions for a user's write access don't change when the semester ends. There must be something else going on. 

I definitely endorse the use of the tilde in code. It will make it more portable to your students, Laura.

Randy