topic Re: x command for copying files not working in SAS Programming

x command for copying files not working

woo — Wed, 23 Jul 2014 17:25:32 GMT

Hello friends - please help

i am using below code to transfer some previous day text files to different folder - but x command is not working - i am running this code in base sas since x command is valid.

filename all_txt pipe 'dir "d:\woo\test\*.txt" /tw';

data transfer_yesterday_files;

attrib buffer format=$2000.

file_last_written_date format=yymmdd10.

file_last_written_time format=time5.

file_name format=$100.;

infile all_txt truncover ;

input buffer $2000.;

/* Remove some extra info */

if substr(buffer,1,5) in ("Volum","Direc") or index(buffer,"<DIR>")>0 or index(buffer,"Dir(s)")>0 or index(buffer,"File(s)")>0 then delete;

file_last_written_date=input(put(scan(compbl(buffer),1,' '),$10.),mmddyy10.); /* note dependant on system settings */

file_last_written_time=input(put(scan(compbl(buffer),2,' '),$5.),time5.);

file_name=trim(scan(compbl(buffer),5,' ')) ;

if file_name = ' ' then delete ;

run;

data _null_;

tempdate1=intnx('day',today(),0);

tempdate2=intnx('day',today(),-1);

tempdate3=intnx('day',today(),-2);

tempdate4=intnx('day',today(),-3);

tempdate5=intnx('day',today(),-4);

tempdate6=intnx('day',today(),-5);

tempdate7=intnx('day',today(),-6);

call symput('date1',put(tempdate1,yymmdd10.));

call symput('date2',put(tempdate2,yymmdd10.));

call symput('date3',put(tempdate3,yymmdd10.));

call symput('date4',put(tempdate4,yymmdd10.));

call symput('date5',put(tempdate5,yymmdd10.));

call symput('date6',put(tempdate6,yymmdd10.));

call symput('date7',put(tempdate7,yymmdd10.));

run;

%put _user_;

data _null_;

set transfer_yesterday_files;

if file_last_written_date=&date6 then

do;

x copy "d:\woo\test\*.txt d:\woo\test\transfer\&date6"; /*here &date6=2014-07-18 (if we run code today, 2014-07-23) and i have daily batch process which creates previous day folder in same format, 2014-07-18*/

end;

run;

Re: x command for copying files not working

Tom — Wed, 23 Jul 2014 18:19:35 GMT

I think one problem lies here:

if file_last_written_date=&date6 ...

It looks like you are trying to compare a data value to the result of subtracting 18 from the difference between 2014 and 7.

Try changing to:

if file_last_written_date=input("&date6",yymmdd10.) ...

Other problem is that it looks like your operating system COPY command will copy all of the .txt files and not just the ones with the date you specified. Is that what you want?

Re: x command for copying files not working

woo — Wed, 23 Jul 2014 18:44:33 GMT

No Tom - you are right - i am not going to copy all text files - just specific text file

filename all_txt pipe 'dir "d:\woo\test\*.txt" /tw';

data transfer_yesterday_files;

attrib buffer format=$2000.

file_last_written_date format=yymmdd10.

file_last_written_time format=time5.

file_name format=$100.;

infile all_txt truncover ;

input buffer $2000.;

/* Remove some extra info */

if substr(buffer,1,5) in ("Volum","Direc") or index(buffer,"<DIR>")>0 or index(buffer,"Dir(s)")>0 or index(buffer,"File(s)")>0 then delete;

file_last_written_date=input(put(scan(compbl(buffer),1,' '),$10.),mmddyy10.); /* note dependant on system settings */

file_last_written_time=input(put(scan(compbl(buffer),2,' '),$5.),time5.);

file_name=trim(scan(compbl(buffer),5,' ')) ;

if file_name = ' ' then delete ;

run;

data _null_;

tempdate1=intnx('day',today(),0);

tempdate2=intnx('day',today(),-1);

tempdate3=intnx('day',today(),-2);

tempdate4=intnx('day',today(),-3);

tempdate5=intnx('day',today(),-4);

tempdate6=intnx('day',today(),-5);

tempdate7=intnx('day',today(),-6);

call symput('date1',put(tempdate1,yymmdd10.));

call symput('date2',put(tempdate2,yymmdd10.));

call symput('date3',put(tempdate3,yymmdd10.));

call symput('date4',put(tempdate4,yymmdd10.));

call symput('date5',put(tempdate5,yymmdd10.));

call symput('date6',put(tempdate6,yymmdd10.));

call symput('date7',put(tempdate7,yymmdd10.));

run;

%put _user_;

%macro trans;

data _null_;

set transfer_yesterday_files;

if file_last_written_date=&date6 then

do;

x "mkdir d:\woo\test\&date6";

x "copy /*i need copy command here which would copy files which are having "file_last_written_date=&date6*/

x "copy "d:\woo\test\*.txt d:\woo\test\&date6";

x "gzip "/*i need gzip command here which will zip all transferred files into newly created folder "&date6" which will be date folder like 2014-07-18*/

end;

run;

%mend;

%trans;

Re: x command for copying files not working

Tom — Wed, 23 Jul 2014 19:18:34 GMT

Depends on what version of GZIP you have installed on your Windows machine. Here is one page that suggests that the standard Unix style might work. ftp://ftp.whrc.org/Pre-LBA/software/gziptar/gzip.txt

If you want a little more control over what the compressed file is named you can use the c option and re-direct the output of gzip to a file. An example illustrates this usage:

C:\>gzip -c thefile.txt > thefile.gz

If you use that syntax then you do not need to first COPY the file. You will probably need to know where the command "gzip" is on your computer as you might need to use the fully qualified path to the program to get it run. For example it might be something like "c:\program files\wingzip\gzip.exe"

You are working too hard to read in the output of the DIR command.

Once you have it the list of files you could then sort by the last written date and process the files that way.

One way to process the commands is to have them in a character variable and use the INFILE PIPE with FILEVAR= option.

%let path = c:\downloads ;

data transfer_yesterday_files;

attrib file_name length=$255 ;

attrib file_size format=comma20. ;

attrib file_last_written_date format=yymmdd10. ;

attrib file_last_written_time format=time5. ;

infile "dir /tw ""&path\*.txt"" " pipe truncover lrecl=3000 ;

input file_last_written_date ??mmddyy10.

file_last_written_time ??time10.

file_size :??comma20.

file_name $255.

;

if file_last_written_date=. then do;

put _infile_;

delete;

end;

run;

proc sort data=transfer_yesterday_files;

by file_last_written_date ;

run;

data cmds;

set transfer_yesterday_files;

by file_last_written_date ;

length cmd $600 dir $300;

keep cmd ;

dir = cats("&path\",put(file_last_written_date,yymmdd10.),'\') ;

if first.file_last_written_date then do ;

cmd=catx(' ','mkdir',quote(trim(dir))) ;

output;

end;

cmd = catx(' ','gzip -c',quote(cats("&path\",file_name))

,'>',quote(cats(dir,file_name,'.gz')));

output;

run;

proc print;

run;

1	mkdir "c:\downloads\2014-04-28\"
2	gzip -c "c:\downloads\cencty.txt" > "c:\downloads\2014-04-28\cencty.txt.gz"
3	mkdir "c:\downloads\2014-06-30\"
4	gzip -c "c:\downloads\Enhanced Dosage.txt" > "c:\downloads\2014-06-30\Enhanced Dosage.txt.gz"
5	mkdir "c:\downloads\2014-07-08\"
6	gzip -c "c:\downloads\pfizer_manifest.txt" > "c:\downloads\2014-07-08\pfizer_manifest.txt.gz"
7	mkdir "c:\downloads\2014-07-23\"
8	gzip -c "c:\downloads\SeriesDataOut.txt" > "c:\downloads\2014-07-23\SeriesDataOut.txt.gz"
9	gzip -c "c:\downloads\SeriesIn.txt" > "c:\downloads\2014-07-23\SeriesIn.txt.gz"

data _null_;

set cmds ;

infile cmd pipe filevar=cmd end=eof ;

do while (not eof);

input;

put _infile_;

end;

eof=0;

run;

Re: x command for copying files not working

woo — Wed, 23 Jul 2014 21:17:40 GMT

This is awesome Tom - it worked fine - just few questions if you got some time...

Is there any way i can stay with date logic to use below mkdir/copy/zip command? why i am asking is because i will have different dates files and i want to treat specific date files only - so basically this process will run daily - for example

- if this sas code (the one you gave me) run today then it supposed to created folder with yesterday's date and copy yesterday's files into yesterday folder and then zip the files

- but what if folder has more than one different date files - how this code will treat it?

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

/*i wanted to stay with this code if i can - but it doesn't seems feasible for me*/

%macro trans;

data _null_;

set transfer_yesterday_files;

if file_last_written_date=&date6 then

do;

x "mkdir d:\woo\test\&date6";

x "copy /*i need copy command here which would copy files which are having "file_last_written_date=&date6*/

x "copy "d:\woo\test\*.txt d:\woo\test\&date6";

x "gzip "/*i need gzip command here which will zip all transferred files into newly created folder "&date6" which will be date folder like 2014-07-18*/

end;

run;

%mend;

%trans;

Re: x command for copying files not working

Tom — Wed, 23 Jul 2014 22:49:31 GMT

The main logic problem with your code is you have created a macro variable with a nice value for using in the name of the directory, but it is not a value that can be easily used in the WHERE clause. In particular if DATE6=2014-07-22 then your directory name D:\WOO\TEST\&date6 will work well, but your IF (date_var = &date6) will not because SAS will interpret 2014-07-22 as (2014-07)-22 which is 08JUN1965 if treated as a date.

If you want to limit which file names you process further just put in a WHERE or IF statement in the proper place.

where file_last_written_date = today()-1 ;

Re: x command for copying files not working

Tom — Wed, 23 Jul 2014 23:25:43 GMT

Let's look at your example code line by line. Note that there is no need for a macro as there is no macro logic happening here.

Lets assume that you created a macro variable named DATE6 with value like 2014-07-22.

And that you want to write into that folder the files from the TRANSFER_YESTERDAY_FILES dataset that were modified on that date.

Note a few things that trip up this type of things.

1) Make sure to use quotes around paths and filenames in your generated windows commands. This will allow you to process directories or files that contain embedded spaces .

2) If you have a literal string in a SAS state that is using double quotes on the outside (so that macro variables are resolved) then you need to double up the embedded double quote characters so that SAS does not interpret them as ending the literal string that you started with a double quote.

3) Use the CALL SYSTEM() command instead of the X command so that you can generate the commands from the data.

data _null_;

set transfer_yesterday_files;

where file_last_written_date = input("&date6",yymmdd8.);

/* You want to use the CALL SYSTEM command instead of the X command */

/* inside of a data step. That way you can dynamically generate the command. */

/* See man pages: SAS(R) 9.2 Companion for UNIX Environments */

* Make a directory ;

call system ("mkdir ""d:\woo\test\&date6"" ");

* Copy the file from source directory to the new directory ;

call system('copy "d:\woo\test\' || trim(file_name) || '"' ||" ""d:\woo\test\&date6\"" ");

* Use GZIP command to compress it.;

* Note that GZIP is NOT a Windows command so you will need to install it if you do not have it ;

* You should test the syntax on your machine in a command window to make you can do this on your machine. ;

call system("gzip ""d:\woo\test\&date6\" || trim(file_name) ||'"');

run;

Re: x command for copying files not working

woo — Thu, 24 Jul 2014 00:03:28 GMT

THANKS A LOT TOM for your sizable time and explanation - this will help me for other logic creation too...

Really appreciate your efforts....

Thanks!