topic Re: Archiving files in SAS using UNIX in SAS Programming

Archiving files in SAS using UNIX

astha8882000 — Thu, 19 Jul 2018 20:15:37 GMT

Hello,

I have a code that I'm trying to correct because I wanted to understand and learn the unix component in sas. The aim is to archive files and move them from the "original path" to the "new path". This is the code that I received:

abc="mv -f /original path" || strip (_infile_) || "/new path";

if not fileexist("/new path" || strip (_infile_)

then rc=system(abc));

put abc=;

abc= 'find /new path -type f \( -iname "*.*" ! -iname "*.gz" ! -iname "*.Z" ! -iname "*.tgz" ! -iname "*.zip" \) -execdir gzip {} \;';

rc=system(abc);

put abc=;;

run;

So far this is what I have understood:

abc="mv -f /original path" || strip (_infile_) || "/new path";--this command moves the files from the old location to the new location, creating the new location folder structure

if not fileexist("/new path" || strip (_infile_) then rc=system(abc));--if the new location directory/folder structure does not exist, then run the command abc to create the new folder structure and move the files

rc=system(abc);--return the value received after the execution of the above command abc

put abc=;--if successful execution, returned value is 0, else 1

abc= 'find /new path -type f \( -iname "*.*" ! -iname "*.gz" ! -iname "*.Z" ! -iname "*.tgz" ! -iname "*.zip" \) -execdir gzip {} \;';--command abc will now switch to the new path directory and execute the specified command.

rc=system(abc);--return the value received after the execution of command abc

put abc=;;--if successful execution, returned value is 0, else 1

run;

Please suggest if my interpretation of the above code lines is correct.

Also, in the line abc= 'find /new path -type f \( -iname "*.*" ! -iname "*.gz" ! -iname "*.Z" ! -iname "*.tgz" ! -iname "*.zip" \) -execdir gzip {} \;'; what does the entire iname command even mean? Is it zipping all the files irrespective of what their names and extensions are? Or are the files already zipped and it is converting that to gzip? I just cannot interpret that.

And I think these are the following syntax errors in this piece of code, please suggest if that's how unix syntax works:

1. Instead of abc="mv -f /original path" || strip (_infile_) || "/new path";

it should be abc=mv -f "/original path" || strip (_infile_) || "/new path";

2. Instead of

if not fileexist("/new path where the files need to go after being archived" || strip (_infile_)

then rc=system(abc));

it should be

if not fileexist("/new path where the files need to go after being archived" || strip (_infile_))

then rc=system(abc);

but isn't filexist supposed to have only one arguement?

3. Just before run, put abc=;;--I'm hoping this is a typo, why would there be 2 ; anyway?

Thank you!

Re: Archiving files in SAS using UNIX

ballardw — Thu, 19 Jul 2018 21:14:45 GMT

Generally along the right path.

So far this is what I have understood:

abc="mv -f /original path" || strip (_infile_) || "/new path";--this command moves the files from the old location to the new location, creating the new location folder structure

Not quite but close, it assigns the TEXT of an operating system command to a text variable so that the actual command can be passed with the SYSTEM function (or CALL SYSTEM). RC then receives the return value of the SYSTEM attempting to execute that command.

1. Instead of abc="mv -f /original path" || strip (_infile_) || "/new path";

it should be abc=mv -f "/original path" || strip (_infile_) || "/new path";

The statement: abc=mv -f "/original path" || strip (_infile_) || "/new path"; would have a syntax error as it would first attempt to subtract the value of a numeric variable f from a numeric variable mv (not a problem unless you actually have those variables) but without an operation the "/original path" will generate an error like:

ERROR 22-322: Syntax error, expecting one of the following: !, !!, &, (, *, **, +, -, /, <, <=,
              <>, =, >, ><, >=, AND, EQ, GE, GT, LE, LT, MAX, MIN, NE, NG, NL, OR, [, ^=, {, |,
              ||, ~=.

2. The syntax for Filename is: FILEEXIST(file-name) ; Filename can be a literal or any expression that resolves to a (hopeful) filename.

if not fileexist("/new path where the files need to go after being archived" || strip (_infile_)

then rc=system(abc));

has an issue that the fileexist is not properly closed I suspect it should have been:

if not fileexist("/new path where the files need to go after being archived" || strip (_infile_) )

then rc=system(abc);

3. Just before run, put abc=;;--I'm hoping this is a typo, why would there be 2 ; anyway?

Why? Cannot be sure. There may have been something deleted or just a double tap on the ; key. An extra ; is a null statement as far as SAS is concerned.

About -iname I have no clue as that looks like a Unix command.

If you run code and get errors that SAS detects then post the code and error messages. Copy the bit from the log and paste into a code box opened using the forum {I} icon to preserve format of text and position of diagnostics that SAS often places in the code indicating where the error was detected.

Re: Archiving files in SAS using UNIX

Kurt_Bremser — Thu, 19 Jul 2018 21:20:19 GMT

-iname is an expression of the find command, see https://linux.die.net/man/1/find

Re: Archiving files in SAS using UNIX

ChrisNZ — Thu, 19 Jul 2018 21:54:34 GMT

0. put abc=; lets you see the command sent, not the return code. You need put RC=; for that.

1. abc=mv -f "/original path" || strip (_infile_) || "/new path"; is invalid syntax.

If you want to quote the paths use abc='mv -f "/original path/' || strip (_infile_) || '" "/new path"';

2. if not fileexist("/new path" || strip (_infile_)) then rc=system(abc);

is correct. Concatenating 2 strings still makes only one argument

3. One semicolon is enough.

Re: Archiving files in SAS using UNIX

Tom — Thu, 19 Jul 2018 22:27:51 GMT

You seem to have left off the top of the program. It looks like you are reading from a file (or perhaps in-line) data since you are referencing the automatic variable _INFILE_.

You will have much better luck using PIPE as then you can see the messages that the operating system generates.

Take advantage of SAS functions to make your code generation easier. Look at the CATxxx series of functions. And the QUOTE() function.

The last step seems to a effort to compress files. The -iname predicates to the find command are telling it ignore file extensions that normally indicate that the file is already compressed. You should probably run that AFTER you have finished moving all of the files. Or just test the name of the file yourself and let SAS decide if it should be gzipped or not.

%let indir=/source/directory/name ;
%let outdir=/target/directory/name ;

data _null_;
   input @ ;
   length from to $200 ext $10 cmd1 cmd2 $500 ;
   from = catx('/',"&indir",_infile_);
   to = catx('/',"&outdir",_infile_);
   if index(_infile_,'.') then ext = scan(_infile_,-1,'.');
  cmd1=catx(' ','mv -f',quote(strip(from)),quote(strip(to)));
  infile cmd pipe filevar=cmd1 end=eof1;
  while (not eof1);
    input;
    put _infile_;
  end;
  if ext not in ('gz','Z','tgz','zip') then do;
    cmd2=catx(' ','gzip',quote(strip(to)));
    infile cmd2 pipe filevar=cmd2 end=eof2;
    while (not eof2);
      input;
      put _infile_;
    end;
  end;
cards4;
archive_me.sas7bdat
;;;;

Re: Archiving files in SAS using UNIX

astha8882000 — Fri, 20 Jul 2018 14:13:21 GMT

You are right, the statement below is giving the error as you mentioned:

1. Instead of abc="mv -f /original path" || strip (_infile_) || "/new path";

it should be abc=mv -f "/original path" || strip (_infile_) || "/new path";
The statement: abc=mv -f "/original path" || strip (_infile_) || "/new path"; would have a syntax error as it would first attempt to subtract the value of a numeric variable f from a numeric variable mv (not a problem unless you actually have those variables) but without an operation the "/original path" will generate an error like:

ERROR 22-322: Syntax error, expecting one of the following: !, !!, &, (, *, **, +, -, /, <, <=,
<>, =, >, ><, >=, AND, EQ, GE, GT, LE, LT, MAX, MIN, NE, NG, NL, OR, [, ^=, {, |,
||, ~=.

What should be the correct syntax for this line of code?