topic How to check if an external file is ready to be imported? in SAS Procedures

How to check if an external file is ready to be imported?

Augusto — Wed, 14 Mar 2012 17:41:38 GMT

Hi everyone,

Is there a way to check if a file is totaly transfered before start importing it?

I am transfering a file to the SAS server and I would run my infile code only when the transfer has finished.

This file has about 50 GB, i tried using this code below BUT IS NOT CORRECT BECAUSE THE FILE have_2012*.TXT is still in transfering and my table WANT WILL NOT HAVE all the records inside.

data _null_;

do until (fileexist("/INBOX/have_2012*.TXT"));

call sleep(5);

end;

run;

data want;

infile "/INBOX/have_2012*.TXT" MISSOVER DSD lrecl=32;

input

@0001 ANO_MVM $004.

@0005 MES_MVM $002.

@0007 COD_AGN $004.

@0011 COD_OPR $013.

@0024 ACERTO_OPR $002.

@0026 COD_CMT_OPR $002.;

run;

thanks for any suggestion.

Augusto Souza

How to check if an external file is ready to be imported?

FriedEgg — Wed, 14 Mar 2012 17:47:10 GMT

How is the file being transfered?

Re: How to check if an external file is ready to be imported?

Augusto — Wed, 14 Mar 2012 17:53:27 GMT

BY COPY AND PASTE, SO I CAN FOLLOW THE FILE SIZE BEING INCREASED BY WinSCP TOOL.

I AM IN WINDOWS O.S.

Re: How to check if an external file is ready to be imported?

SASKiwi — Wed, 14 Mar 2012 19:01:16 GMT

I haven't tried it myself but how about:

rc=filename('myfile',fname);

fid=fopen('myfile');

I would imagine the file open would fail because it would still be locked while it is being transferred, particularly if you opened the file for update.

How to check if an external file is ready to be imported?

FriedEgg — Wed, 14 Mar 2012 19:44:38 GMT

What is the server you are transfering to? Why are you starting the SAS process before the transfer has started in the first place?

How to check if an external file is ready to be imported?

ballardw — Wed, 14 Mar 2012 21:28:15 GMT

If you system and the server are running with the same (or close enough time stamps) you might be able to use the DOPEN, DOPTNUM and DINFO functions to read time stamps of the given file until it looks like the time stamps quit updating.

How to check if an external file is ready to be imported?

Augusto — Thu, 15 Mar 2012 02:16:08 GMT

Hi Guys,

Firstly, sorry for my english I am from Brazil, sencond thanks so much for the answers.

Well, It has already transfered and I ran my process thanks.

Welll, I imagined something like, to check the file size. As I mentioned before, this file has about 50 GB, I don't know if this file is locked even when its being transfered, because I can import it but not all the records. In my machine i have windows xp and the SAS is installed in unix. Through WinSCP in windows I can follow the transfer process until its finished.

I am not sure, but maybe I could check it through unix commands (any suggestion?).

if anyone knows any way to check it, It will be greatly appreciated.

Thanks

Augusto Souza.

Re: How to check if an external file is ready to be imported?

Lulu_W — Thu, 15 Mar 2012 04:00:24 GMT

I haven't tried myself, but how about using some scripts (like windows batch command) to copy and paste the file and create a dummy file (like success.txt) when the transfer process is finished, so you can check the dummy file to see if the transfer is completed and if it is there start reading; else wait for a while and repeat the process.

Re: How to check if an external file is ready to be imported?

SASKiwi — Thu, 15 Mar 2012 19:29:31 GMT

Did you try the FOPEN while the file was transferring? Using the 'U' option will try to open it for update which should fail if the file is still being written.

data _null_;

rc=filename('myfile',fname);

fid=fopen('myfile','U');

put rc = fid = ;

run;

If fid = 0 then you know the open failed. Also check that rc = 0 to ensure the filename function worked.

Re: How to check if an external file is ready to be imported?

Ksharp — Fri, 16 Mar 2012 09:06:24 GMT

or maybe can use PIPE to calcualted the size of these files after very Five minutes .

If there is no change ,then use the code to import.

Ksharp

How to check if an external file is ready to be imported?

Augusto — Fri, 16 Mar 2012 17:37:50 GMT

Ksharp,

Could you post an example of using pipe?

Thanks

How to check if an external file is ready to be imported?

Augusto — Fri, 16 Mar 2012 17:42:54 GMT

Thanks SASkiwi. I'll make the try.

How to check if an external file is ready to be imported?

Ksharp — Mon, 19 Mar 2012 02:41:54 GMT

OK. Here is an example, if your site can use PIPE. I copy all the tables at a directory into WORK library.

%macro sleep_copy;
%local size pre_size;
%do %until(&pre_size=&size);
%let pre_size=&size;
filename x pipe 'dir c:\test\*.sas7bdat';
data _null_;
call sleep(200);
infile x dlm=' ' expandtabs end=last;
input date ?? : yymmdd12. time ?? : time6. size ?? : comma32. name : $40.;
total_size+size;
if last then call symputx('size',total_size);
run;
%end;
libname x v9 'c:\test';
proc  copy in=x out=work memtype=data;
run;
%mend;


%sleep_copy

Ksharp

How to check if an external file is ready to be imported?

Augusto — Mon, 19 Mar 2012 14:45:58 GMT

Thanks so much Ksharp.

Augusto Souza