topic Re: Bulk Download all my files from SAS Ondemand in SAS Software for Learning Community

Bulk Download all my files from SAS Ondemand

LauraRK — Wed, 20 Jan 2021 19:35:36 GMT

I taught my class last semester using SAS Ondemand and now I want to download all the files we used to archive them. Please tell me I do not have to click on each file one at a time. I have spent time looking but have yet to find away to download all my files from SAS Ondemand to my local machine without clicking each one. My question is, how can one download all ones files from SAS Ondemand for Academics without having to download them one at a time? I am a professor and I was using SAS studio.

Re: Bulk Download all my files from SAS Ondemand

RandyMullis — Wed, 20 Jan 2021 20:36:05 GMT

Hi, @LauraRK . I hate to be the bearer of bad news. There isn't a facility to do this right now. I'm sorry.

Randy

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Thu, 21 Jan 2021 11:59:00 GMT

Thank you Randy.

I asked in the programming section if there is away to zip directories and keep directory structures. Given I am teaching I set up examples and exercises in lots of folders. I wish I had realized this earlier as I would never have used SAS OnDemand to set up the class. I would have just kept everything local. This is a pretty big workflow challenge.

Re: Bulk Download all my files from SAS Ondemand

RandyMullis — Thu, 21 Jan 2021 12:03:35 GMT

I agree, Laura, it's a big need. The operations team tells me it's on their futures list. I think it would be a great challenge for a good SAS programmer. I've seen several programs that do this, but not in the kind of robust way that would be generally usable. I'm sorry this is an obstacle for you. Best to you in your upcoming teaching.

Randy

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Thu, 21 Jan 2021 12:53:12 GMT

I think other potential ways would be github integration or integrate with dropbox, or someway a person can save files and have a project workflow and then download that work. One of the biggest challenges in teaching and programming is workflow.
Minimally, I think SAS should have a warning that one should not plan on easily downloading their files in the set up documents for SAS ondemand.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Thu, 21 Jan 2021 13:54:11 GMT

What kind of files to you need to add to the ZIP archive? If it's only SAS programs (text), there might be a not-so-complicated way.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Thu, 21 Jan 2021 14:02:09 GMT

I just cobbled together a method to create an archive that contains all program files in a directory tree.

First, I used a program created by @Tom that is a very ingenious way of recursively scanning a directory tree in one data step:

data filelist;
  length dname filename $256 dir level 8 lastmod size 8;
  format lastmod datetime20.;
  input dname;
  retain filename ' ' level 0 dir 1;
cards4;
/folders/myfolders
;;;;

data filelist;
  modify filelist;
  rc1=filename('tmp',catx('/',dname,filename));
  rc2=dopen('tmp');
  dir = not not rc2;
  if not dir then do;
    fid=fopen('tmp','i',0,'b');
    lastmod=input(finfo(fid,foptname(fid,5)),NLDATM100.);
    size=input(finfo(fid,foptname(fid,6)),32.);
    fid=fclose(fid);
  end;
  else do;
    dname=catx('/',dname,filename);
    filename=' ';
    lastmod=input(dinfo(rc2,doptname(rc2,5)),NLDATM100.);
  end;
  replace;
  if dir;
  level=level+1;
  do i=1 to dnum(rc2);
    filename=dread(rc2,i);
    output;
  end;
  rc3=dclose(rc2);
run;

Then, I used the resulting dataset to run a macro repeatedly that adds members to a ZIP archive:

filename out zip "/folders/myfolders/all.zip";

%macro write_single(fname);

data _null_;
infile "&fname";
file out("&fname");
input;
put _infile_;
run;

%mend;

data _null_;
set filelist;
where filename ne "" and scan(filename,-1,".") = "sas";
call execute('%write_single('!!catx("/",dname,filename)!!")");
run;

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Thu, 21 Jan 2021 14:01:49 GMT

If I can minimally do the SAS files and ideally also the directory structure that would be great. I know I have spent more time on this than I would have spent just doing this by hand one at a time, but if I am going to keep using the platform I need to figure out a workflow. I also would love to be able to provide this tool to my students, so they can zip their work after they are done with the class. Thank you so much.

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Thu, 21 Jan 2021 14:37:04 GMT

For some reason I could not get it to work on SAS ondemand?

It runs and I get no errors but then my zip file is empty. I do get the file list. I do appreciate so much your help. I have some meetings for the rest of the day but I will work on this more later tonight or tomorrow, and try it locally as well.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Tue, 26 Jan 2021 08:46:38 GMT

I just did a test run on my On Demand account; I only replaced the base paths (/folders/myfolders) with the path to my On Demand home directory, and successfully created a zip file with the .sas program files that I have there.

Everything worked fine, I could extract the two .sas files after downloading to my laptop.

Please take a close look at your log; for each .sas file found, you should have this from the ZIP step:

 NOTE: CALL EXECUTE generated line.
 1         + data _null_; infile "/home/XXX/programs/zip_all.sas"; file out("/home/XXX/programs/zip_all.sas"); 
 input; put _infile_; run;
 
 NOTE: The infile "/home/XXX/programs/zip_all.sas" is:
       Dateiname=/home/XXX/programs/zip_all.sas,
       Besitzername=XXX,Gruppenname=oda,
       Zugriffsberechtigung=-rw-r--r--,
       Zuletzt geändert=26. Jänner 2021 09.39 Uhr,
       Dateigröße (Byte)=312
 
 NOTE: The file library OUT is:
       Directory=/home/XXX/all.zip
 
 NOTE: The file OUT("/home/XXX/programs/zip_all.sas") is:
       Filename=/home/XXX/all.zip,
       Member Name=/home/XXX/programs/zip_all.sas
 
 NOTE: 18 records were read from the infile "/home/XXX/programs/zip_all.sas".
       The minimum record length was 0.
       The maximum record length was 62.
 NOTE: A total of 18 records were written to the file library OUT.
       The minimum record length was 0.
       The maximum record length was 62.
 NOTE: 18 records were written to the file OUT("/home/XXX/programs/zip_all.sas").
       The minimum record length was 0.
       The maximum record length was 62.

(home path redacted)

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Tue, 26 Jan 2021 21:08:48 GMT

Thank you so much. I got it to work once I moved the files out of my shared folder and into my home directory. I had the correct paths but I think with the shared drive there is some special protection because I was getting a very odd error.

This is very helpful and I appreciate your work on this very much.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Wed, 27 Jan 2021 08:43:15 GMT

Could you show the "very odd error"? This might enable us to point to a solution that does not require moving all the files.

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Wed, 27 Jan 2021 13:18:59 GMT

I should have copied it at the time. When I restarted SAS Ondemand it no longer was there, but I also no longer had write access to my shared drive. I think that perhaps what happens is that since my class ended maybe my permissions to my drive changed in the last couple days?

What I did figure out is that it will work with the shared drive as long as I use the ~ and do not use the copied path.

If I use the below:

data filelist;
  length dname filename $256 dir level 8 lastmod size 8;
  format lastmod datetime20.;
  input dname;
  retain filename ' ' level 0 dir 1;
cards4;
 /home/u49xxxxxx/my_shared_file_links/kxxxxxx/sta216
;;;;

my zip folder is empty. Although, it does create a listing of the files in the folders they do not end up copied into the zip file.

However, if I do:

data filelist;
  length dname filename $256 dir level 8 lastmod size 8;
  format lastmod datetime20.;
  input dname;
  retain filename ' ' level 0 dir 1;
cards4;
~/my_shared_file_links/kxxxxxx/sta216
;;;;

It completely works. So your solution works great. Next time I will make sure I do this while I still have proper access to my drives. This was my first semester using SAS OnDemand, I am an experienced SAS user but there was a bit of a learning curve with using drives that I could not access directly outside of SAS. I very much appreciate this exercise, and your help on this Kurt. This is nice because I can give this tool to my students to run as well.

My next question is can I do this in reverse? It seems I can, but I would need to have sas create the directories as I unzipped. Perhaps using a libname and dlcreatedir option? I also would love to be able to upload zipped data as well, not just SAS programs and unzip it on SAS Ondemand but I do not see that option yet that I can find.

Thanks so much for your help. I am very appreciative.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Wed, 27 Jan 2021 13:43:49 GMT

Maybe they did something weird with the contents of $HOME. The tilde returns the contents of that environment variable.

Re: Bulk Download all my files from SAS Ondemand

RandyMullis — Wed, 27 Jan 2021 14:10:40 GMT

Hi, @Kurt_Bremser and @LauraRK . The permissions for a user's write access don't change when the semester ends. There must be something else going on.

I definitely endorse the use of the tilde in code. It will make it more portable to your students, Laura.

Randy

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Wed, 27 Jan 2021 14:38:26 GMT

I am very embarrased but I realized what happened, I have two accounts, one I use to pretend to be a student, and one is my actual teacher account. I was accidentally logged into the wrong account. Nevertheless, it is somewhat of a fortunate error as it allows me to now have a more robust solution and to understand a bit better the file system for SAS Ondemand. It also helps me to be more empathetic, and be reminded that it is easy to make small mistakes when one is thinking about harder issues and does not take a few moments to be mindful of what is going on. I also put a bunch of time into learning Github and integrating it with R-studio cloud, so I think my next goal might be figuring that out with SAS Ondemand, https://blogs.sas.com/content/sasdummy/2019/01/17/git-in-sas/ . @RandyMullis do you know if that is possible with SAS Ondemand?

Meanwhile, I think the tool @Kurt_Bremser has given us is very nice and I will definitely use it to zip and download my SAS programs in nice neat directories.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Wed, 27 Jan 2021 14:53:36 GMT

I have made it a habit to always use $HOME for the home directory. That makes code portable.

And not only that. Anything you can get from environment or automatic macro variables, or other means (e.g. functions) should be defined that way.

Re: Bulk Download all my files from SAS Ondemand

Tom — Wed, 27 Jan 2021 15:02:50 GMT

So you want the companion UNZIP macro to go with the ZIP method in the selected solution?

You could probably remove the references to %PARMV() macro or get it from github.

%macro unzip
/*----------------------------------------------------------------------------
Extract files from a ZIP file using only SAS code
----------------------------------------------------------------------------*/
(zipfile    /* Source file */
,outdir     /* Target directory */
,dsname=unzipfiles /* Name of dataset created with filenames */
,createdir=YES  /* Use DLCREATEDIR option to make subdirectories */
);
%local macro parmerr dlcreatedir;
%let macro=&sysmacroname;
%let dlcreatedir=%sysfunc(getoption(dlcreatedir));
%parmv(zipfile,_req=1,_case=n,_words=1)
%parmv(outdir,_req=1,_case=n,_words=1)
%parmv(dsname,_case=n)
%parmv(createdir,_val=0 1)
%if (&parmerr) %then %goto quit;

*-----------------------------------------------------------------------------;
* Get list of members in the ZIP file ;
*-----------------------------------------------------------------------------;
data &dsname ;
  length fileref $8 fileno 8 memname zipfile $256.;
  zipfile = symget('zipfile');
  rc=filename(fileref,zipfile,'zip');
  if rc then call execute('%nrstr(%parmv)(zipfile,_msg=Unable to open file)');
  else do;
    did=dopen(fileref);
    if not did then call execute
       ('%nrstr(%parmv)(zipfile,_msg=Unable to open file as a zip file)')
    ;
    else do;
      do fileno=1 to dnum(did);
        memname=dread(did,fileno);
        output;
      end;
      did=dclose(did);
      rc=filename(fileref);
    end;
  end;
  drop fileref did rc;
run;
%if (&parmerr) %then %goto quit;

*-----------------------------------------------------------------------------;
* Copy members to files in target directory ;
*-----------------------------------------------------------------------------;
%if (&createdir) %then %do;
options dlcreatedir;
%end;
%else %do;
options nodlcreatedir;
%end;
data &dsname ;
  set &dsname ;
  length libref in out $8 outfile $256.;
  retain libref ' ';
  if "&sysscp"='WIN' then sep='\';
  else sep='/';
  rc1=filename(in,zipfile,'zip'
              ,catx(' ','recfm=f lrecl=512','member=',quote(trim(memname))))
  ;
  outfile=symget('outdir');
  rc2=libname(libref,outfile);
  do level=1 to countw(memname,'/')-1;
    outfile=catx(sep,outfile,scan(memname,level,'/'));
    rc2=libname(libref,outfile);
  end;
  rc2x=libname(libref);
  outfile=catx(sep,outfile,scan(memname,level,'/'));
  rc3=filename(out,outfile,,'recfm=f lrecl=512');
  rc4=fcopy(in,out);
  length msg $ 384;
  if rc4 then do;
     msg=sysmsg();
     call symputx('parmerr','1');
     put msg;
  end;
  drop libref in out ;
run;
%if (&parmerr) %then %do;
  %parmv(_msg=Not all files created);
%end;
%quit:
options &dlcreatedir ;
%mend unzip;

Re: Bulk Download all my files from SAS Ondemand

LauraRK — Wed, 27 Jan 2021 15:14:25 GMT

So use $HOME instead of ~ . Just to make sure I understand. Thanks.

Re: Bulk Download all my files from SAS Ondemand

Kurt_Bremser — Wed, 27 Jan 2021 18:06:50 GMT

Both will work, but I think the use of $HOME predates the tilde. Since I have almost 4 decades of working with UNIX, call me old school.