topic Re: Copying multiple files from 1 drive to another in SAS Programming

Copying multiple files from 1 drive to another

BIDD — Wed, 22 Aug 2018 11:37:29 GMT

Hi All,

Is there any syntax without using X Copy to move files from 1 drive to another? Due to security reason IT security advised me not to use windows based command ( X copy). I have a new requirement now where in I have to transfer over 100 different formatted files (rtf,xls,txt,csv) in 1 go from 1 drive to another.

I did google search and got this code, this code is creating copies in same folder by adding _new to the original file names.

Not sure how to amend this code to dump all the files to in to another drive.

let source=P:\Projects\Submissions\DEET18;
%macro directorylisting

  ( path   =   , outdsn = dirlist  , where =     , after  = 01Jan1960  ) ;

  data &outdsn. ( keep = filename fullfilename created modified bytes ) ;
    attrib

      dref         length = $8

      fref         length = $8

      folder       length = $256

      filename     length = $128 label = 'Filename'

      fullfilename length = $256 label = 'Full Filename'

      created      length = 8    label = 'Created'   format = datetime19.

      modified     length = 8    label = 'Modified'  format = datetime19.

      bytes        length = 8    label = 'Bytes'     format = comma15. ;


    if fileexist( "&path." ) then do ;

      folder = ifc( substr( "&path.", lengthn( "&path." ), 1 ) = "\"

        , "&path.", cats( "&path.", "\" ) ) ;

      rc     = filename( dref, folder );

      did    = dopen( dref ) ;

      dcount = dnum( did ) ;


      do i = 1 to dcount ;

        filename = dread( did, i ) ;

      

        if find( filename, "&where.", 'i' ) > 0 then do ;

          fullfilename = cats( folder, filename ) ;

          rc  = filename( fref, fullfilename ) ;

          fid = fopen( fref ) ;

          modified = input( finfo( fid, 'Last Modified' ), anydtdtm. ) ;

          if datepart( modified ) >= "&after."d then do ;

            created = input( finfo( fid, 'Create Time' ), anydtdtm. ) ;

            bytes   = input( finfo( fid, 'File Size (bytes)' ), 18. ) ;

            output ;

          end ;

          fid = fclose( fid ) ;

          rc  = filename( fref, '' ) ;

        end ;

      end ;


      did = dclose( did ) ;

      rc  = filename( dref, '' ) ;

    end ;

    else put "ERROR: The folder &path. does not exist." ;

  run ;

%mend ;
 
%directorylisting(path=&source);

%macro aa;

 data _null_; 

  set dirlist; 

  pos1=index(filename,'.');

  fname1="&source"||'\'||filename; 

  new_fname="&source"||'\'||substr(filename,1,pos1-1)||'_new'||substr(filename,pos1);   

  call symput('file'||strip(_n_), '"'||strip(fname1)||'"'); 

  call symput('new_file'||strip(_n_),'"'||strip(new_fname)||'"');

  call symput('cnt1',_n_);

run;

%do ii = 1 %to &cnt1;

     %put &&file&ii=;

   %put &&new_file&ii=;
   /* these IN and OUT filerefs can point to anything */

    filename in &&file&ii;

    filename out &&new_file&ii;
   /* copy the file byte-for-byte  */

  data _null_;

    length filein 8 fileid 8;

    filein = fopen('in','I',1,'B');

    fileid = fopen('out','O',1,'B');

    rec = '20'x;

    do while(fread(filein)=0);

      rc = fget(filein,rec,1);

      rc = fput(fileid, rec);

      rc =fwrite(fileid);

    end;

    rc = fclose(filein);

    rc = fclose(fileid);

  run;

  filename in clear;

  filename out clear;
  
%mend;


%mend aa;

%aa;

Your help is much appreciated.

Re: Copying multiple files from 1 drive to another

Kurt_Bremser — Wed, 22 Aug 2018 10:46:34 GMT

xcopy is part of the Windows operating system. If you don't trust it, get rid of Windows altogether.

Your so-called "security guys" seem to belong to the kind of people that Torvalds ranted about quite recently: https://plus.google.com/+LinusTorvalds/posts/PeFp4zYWY46

Re: Copying multiple files from 1 drive to another

RW9 — Wed, 22 Aug 2018 10:50:12 GMT

Ask your IT security group to securely copy the files across for you. Its not really the job of analysis software to do operating system functions. Thats not to say you can't, but I wouldn't advise it:

https://blogs.sas.com/content/sasdummy/2011/06/17/how-to-use-sas-data-step-to-copy-a-file-from-anywhere/

You of course would then want to validate the copy to ensure it is identical etc. Whereas IT will have copy tools (life teracopy) which check the crc of the two files to ensure they are the same.

In much the same reason as your IT suggested it could be a security risk sending out OS commands (and I can understand that as copying the wrong files, or doing other inadvertent actions could damage your system), doing them through a tool not designed for it is also risky.

Re: Copying multiple files from 1 drive to another

SimonDawson — Wed, 22 Aug 2018 11:30:56 GMT

@ChrisHemedinger wrote a blog post a few years back with a data step that could copy files. Simply wrap this in a macro and job is done.

https://blogs.sas.com/content/sasdummy/2011/06/17/how-to-use-sas-data-step-to-copy-a-file-from-anywhere/

A caveat with this is it will be slow. The reads/writes are done byte by byte. For relatively small files this is a perfectly usable method to move the data around though.

Re: Copying multiple files from 1 drive to another

BIDD — Wed, 22 Aug 2018 11:39:44 GMT

Hi,

I tried my best to educate them, but its of no use. only option left for me now is to amend it.

Re: Copying multiple files from 1 drive to another

BIDD — Wed, 22 Aug 2018 11:43:59 GMT

HI,

I will try this tom and will let you know. Thanks

Re: Copying multiple files from 1 drive to another

Kurt_Bremser — Wed, 22 Aug 2018 11:49:12 GMT

Well, if they only let you have a pocketknife instead of a spanner to change the wheels, let them do it and have all the fun.

It's my favorite method of disarming know-nothing loudmouth idiots: "If you know better, then show me!"

I really relish the ensuing sound of silence, and the sheepish looks.

Re: Copying multiple files from 1 drive to another

ChrisHemedinger — Wed, 22 Aug 2018 12:20:33 GMT

SAS 9.4 also supports the FCOPY function, which might do the job. You still need to assemble the list of source/target files, but it's better than the byte-by-byte copy.

Re: Copying multiple files from 1 drive to another

Kurt_Bremser — Wed, 22 Aug 2018 12:55:48 GMT

Another question: do you have to do it in SAS?

Usually, such tasks should be handled by central scheduling and the datacenter people running it. If you do the selection logic in SAS, write a list to a flat file and let the DC people handle the copy/move.

Re: Copying multiple files from 1 drive to another

Reeza — Wed, 22 Aug 2018 15:23:14 GMT

1. Get a list of files that need to be copied, stored in a data set

2. Use FCOPY to copy the file from one location to another

Step 1 is trivial and I think you can modify the code yourself, if not, check the SAS Macro appendix for a program to list all files in a folder.
Then use FCOPY to move the files over within a data step.

data move_demo;
set file_list; *list of files to be moved;

dir1='path to directory where files are stored'; 
dir2='path to directory where files are to be moved';


rc = fcopy(catx('\', dir1, filename1), catx('\', dir2, filename1));


run;