topic Re: What is the fastest and saves way to copy files without x command in SAS Programming

What is the fastest and saves way to copy files without x command

Faruk — Fri, 29 Apr 2022 09:43:11 GMT

Hoi,

I want to use the folowing method to copy .egp and .sas files from source folder (sas grid) to destination folder (sas grid).

Is this the fastest and the saves way to copy files (egp and sas) to destination folder without corrupting the source and destination files.

Is this bullet proof, so when a developer opens the egp file when this macro is executing, etc.

%macro copy_files(destdir=,totobs=,type=);

%if &totobs. > 0 %then %do;
%do i = 1 %to &totobs.;
data _null_;
TempPath= "&&&copy_&type._&i.";
InName = reverse(TempPath);
InName = substr( InName, 1, index( InName, '/') -1 );
InName = left( reverse( InName ) );

call symputx( 'Inname', Inname );
run;

%put %str(N)OTE: [ Copy from: &&&copy_&type._&i. ];
%put %str(N)OTE: [ Copy to: &destdir/&&Inname. ];

filename insource "&&&KOPIE_&type._&i.";
filename outsource "&doeldir/&&Inname.";

/* Byte for byte copy file */
data _null_;
length filein 8 fileid 8;
filein = fopen('insource','I',1,'B');
fileid = fopen('outsource','O',1,'B');
rec = '20'x;
do while(fread(filein)=0);
rc = fget(filein,rec,1);
rc = fput(fileid, rec);
rc =fwrite(fileid);
end;
rc = fclose(filein);
rc = fclose(fileid);
run;

filename insource clear;
filename outsource clear;

%end;
%end;

%mend copy_files;

Before i execute the macro, ik created a table with list of files in folders. The results of te table ar used to create variables that i passed to the macro. So assume that the work table exists with these rows as example:

enviroment	dir	sourcefile	fullpath	dirniveau	label
O	/srv/SAS/data/source/dev	project_a.egp	/srv/SAS/data/source/dev/project_a.egp	1	dev_source
O	/srv/SAS/data/source/dev	project_b.egp	/srv/SAS/data/source/dev/project_b.egp	1	dev_source
O	/srv/SAS/data/source/prod	project_a.sas	/srv/SAS/data/source/dev/project_a.sas	1	prod_source
O	/srv/SAS/data/source/prod	project_c.egp	/srv/SAS/data/source/dev/project_c.egp	1	prod_source
O	/srv/SAS/data/source/prod	project_c.sas	/srv/SAS/data/source/dev/project_c.sas	1	prod_source

%if (&TABLE_EXISTS eq 1) %then
%do;

proc sql noprint;
select fullpath
into :COPY_SOURCEFILE_1 - :COPY_SOURCEFILE_999999
from WORK.FILELIST;
%let SOURCEFILEOBS = &SYSNOBS.;
quit;
%put &SOURCEFILEOBS. files are found;

%put %str(NOTE: FILELIST existst and files are copied...);

%copy_files(destdir=/srv/destinationdir, totobs=&SOURCEFILEOBS, type=SOURCEFILE);

%end;

Re: What is the fastest and saves way to copy files without x command

Kurt_Bremser — Fri, 29 Apr 2022 10:51:37 GMT

Look at the FCOPY Function

The documentation contains an example for a binary copy.

Re: What is the fastest and saves way to copy files without x command

Faruk — Fri, 29 Apr 2022 18:08:40 GMT

Ok, but why is fcopy better?

Re: What is the fastest and saves way to copy files without x command

Kurt_Bremser — Fri, 29 Apr 2022 18:20:58 GMT

First of all, it saves you a lot of time writing the code. Second, you can bet that SAS has spent a lot of work optimizing the function.

Re: What is the fastest and saves way to copy files without x command

Faruk — Fri, 29 Apr 2022 18:31:18 GMT

ok but i am curious about the difference, why is fcopy better than byte for byte copy.

is fcopy a method that often preffered to copy files? is it faster and are there possibilities that the source can be corrupted by using this procedure?

Re: What is the fastest and saves way to copy files without x command

Kurt_Bremser — Fri, 29 Apr 2022 18:38:25 GMT

As long as you follow the documentation, I see no risk of data corruption.

Regarding the relative speed: try it out.

And if speed is so important to you, use the command provided by the operating system.

Re: What is the fastest and saves way to copy files without x command

Faruk — Fri, 29 Apr 2022 18:57:37 GMT

Ok i will try it out and compare the speed of both.

Most important is that it doesnt corrupt the source and copied files.

the reason why the speed is important is that o copie hundreds of files and it takes 45mnts to copy all the files.

It is not possible to use the command provided by the operating system, because it has been turned off by the administrators for security reasons.

Re: What is the fastest and saves way to copy files without x command

Faruk — Wed, 04 May 2022 13:24:58 GMT

I used:

filename insas "&&&KOPIE_&type._&i." recfm=n;
filename oussas "&doeldir/&&Inname." recfm=n;

data _null_;
length msg $ 384;
rc=fcopy('insas', 'oussas');
if rc=0 then
put 'Copied SRC to DEST.';
else do;
msg=sysmsg();
put rc= msg=;
end;
run;

filename insas clear;
filename oussas clear;

instead of de byte for byte copy. It is way faster, i copied 379 source files (.sas, and .egp) within a minute.

How do i know of a file is binary or not. I used now the binary method?

I am gonna test next week more, by opening a egp file before i copy and also open 1 when i copy to check if anything is blocking or being corrupted.