scenario

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 14
Accepted Solution

scenario

hi all,

i have a scenario where a dataset is copied from source to destination (done by client) and thereafter move from destination to another destination ( i have to do this).

i am trying to automate this . usually these are a huge volume data hence takes time to copy and also that there is no specific time that client copies these dataset(i have no control over it) and when done they are intimated by mail that the copying is started.

The problems i will face is :

i am not sure when the datset copy is completed it take close to 25 mins or more. so if i do copy these before the finish it will be a corrupted dataset.

any idea how i can automate this using sas code .


Accepted Solutions
Solution
‎07-11-2013 11:45 PM
Super Contributor
Posts: 297

Re: scenario

Posted in reply to sugeshnambiar

My most humble of apologies, I obviously didn't read your post properly and assumed incorrectly that you file was an external file and not a SAS dataset.  This is how I deal with SAS datasets:

LIBNAME DROPZONE "E:\DROPZONE";

LIBNAME MOVELOC "E:\";

LIBNAME TEMP "E:\TEMP";

OPTIONS SYMBOLGEN MPRINT MLOGIC FULLSTIMER;

/*CREATE A LARGE FILE FOR TEST MOVING AND PLACE IN TEMP LOCATION*/

DATA TEMP.HAVE;

ARRAY VARS $200 VARS1 - VARS100;

DO J = 1 TO 1000000;

  DO I = 1 TO 100;

  VARS{I} = CATS("STUFF",CEIL(RANUNI(0)*100000000));

  END;

  OUTPUT;

END;

DROP I J;

RUN;

/*MOVE THE FILE MANUALLY TO E:\DROPZONE AND THEN EXECUTE THE FOLLOWING WHILE THE DATASET IS LOCKED TO TEST*/

%MACRO MOVEFILE (INLIB,DSNAME,OUTLIB);

%IF %SYSFUNC(EXIST(&INLIB..&DSNAME.)) %THEN %DO; /*CHECKS TO SEE IF THE DATASET EXISTS*/

   %LET DSID = %SYSFUNC(OPEN(&INLIB..&DSNAME.));  /*IF THE FILE EXISTS IT ATTEMPTS TO OPEN THE DATASET.  IF THE DATASET CAN BE OPENED THEN IT ALLOCATES A DSID */

   %IF &DSID. > 0 %THEN %DO; /*IF A DSID HAS BEEN ALLOCATED THEN CLOSES THE FILE, SO THERE IS NO LOCK ON THE FILE AND MOVES THE DATASET TO THE NEW LOCATION*/

      %LET CLOSE = %SYSFUNC(CLOSE(&DSID.));

      PROC COPY IN=&INLIB. OUT=&OUTLIB. MOVE;

        SELECT &DSNAME.;

    RUN;

   %END;

%END;

%MEND;

%MOVEFILE(DROPZONE,HAVE,MOVELOC); /*SPECIFY THE LIBRARY WHERE THE DATASET IS LOCATED, THE DATASET NAME AND THE LIBRARY THE DATASET SHOULD BE COPIED TO*/

This way the file will not be moved until there is no longer a lock on the dataset, hence reducing the chance of a file corruption.

I hope this is more beneficial.

Regards,

Scott

View solution in original post


All Replies
Super User
Posts: 10,023

Re: scenario

Posted in reply to sugeshnambiar

did you try X statement ?

x 'copy .....' ;

Occasional Contributor
Posts: 14

Re: scenario

i can copy the dataset using
proc dataset;

copy in= out=;

run;

what if the copying of the dataset in not completed and we trigged the above proc, if am not wrong we would be copying a corrupted dataset?

if there a way to check if the dataset is valid and then therafer copy ?

Frequent Contributor
Posts: 118

Re: scenario

Posted in reply to sugeshnambiar

Hi,

If you are using Procedures to copy then you should try with Proc copy as this is faster than Proc dataset.

* What does Valid mean for data set.

Proc copy in=Primary out=secondary;

select DSN;

run;

Ksharp : I am looking for more info on X statement..

Regards

Uma Shanker Saini

Super Contributor
Posts: 644

Re: scenario

Posted in reply to sugeshnambiar

One way to handle this scenario is to have your partner create a trigger file (a small text file - can have any or no content) which gets copied at the end of the main copy job.  At your end, if you have access to a scheduler, it should look for the trigger file every x minutes: when it sees it, it should run a batch job to run your SAS copy program.  At the end of your program you just delete the trigger file and wait for it to come back again.

If you do not have access to a scheduler you can run a continuous SAS job to act as a scheduler.

The main message is that if this copy job is mission critical you need a robust process to manage it.

If you can't get co-operation on the trigger file then writ a SAS jobto test the file size every 5 minute.  If the reading is static you could assume the dopy was finished and kick off your own copy job.

Richard

Occasional Contributor
Posts: 14

Re: scenario

Posted in reply to RichardinOz

@RichardinOZ :thanks for the answer , i shall take that idea...... Smiley Happy

Super User
Posts: 10,023

Re: scenario

Posted in reply to sugeshnambiar

or you can use sleep() to wait one or two then run your proc copy code .

Super Contributor
Posts: 297

Re: scenario

Posted in reply to sugeshnambiar

This is how I deal with this issue:

DATA _NULL_;

IF FILEEXIST("E:\REPORT1.RTF") THEN DO;

    SLEPT = SLEEP(60*30,1);

    X 'MOVE /Y E:\REPORT1.RTF E:\REPORTS\' ;

  END;

RUN;

The FILEEXIST checks to see if the file is in the drop location if so then it waits 30 minutes before the move is executed, which you will have to change to accommodate how long the file typically takes to copy.  All you have to do is set up a job in your scheduler to run every hour.

I hope this is useful.

Regards,

Scott

Super Contributor
Posts: 644

Re: scenario

Posted in reply to sugeshnambiar

Some more ideas

  • If you are using Enterprise Guide as your front end it has a built in scheduler
  • You may be able to use the FINFO() function to test the condition of the file (after using EXIST() fro a SAS table).  Depending on your operating system you could maybe use Last Modified or File size on a Windows platform, or Access Permission under Unix (if the first copy function locks the file) or again look for a change in file size

Richard

Occasional Contributor
Posts: 14

Re: scenario

Posted in reply to RichardinOz

@RichardinOz and @Scott_Mitchell    : you guys rock.... perfect solution..... thanks for the help :smileygrin:

Super Contributor
Posts: 297

Re: scenario

Posted in reply to sugeshnambiar

Glad we could help.

Thanks for your appreciation.

Solution
‎07-11-2013 11:45 PM
Super Contributor
Posts: 297

Re: scenario

Posted in reply to sugeshnambiar

My most humble of apologies, I obviously didn't read your post properly and assumed incorrectly that you file was an external file and not a SAS dataset.  This is how I deal with SAS datasets:

LIBNAME DROPZONE "E:\DROPZONE";

LIBNAME MOVELOC "E:\";

LIBNAME TEMP "E:\TEMP";

OPTIONS SYMBOLGEN MPRINT MLOGIC FULLSTIMER;

/*CREATE A LARGE FILE FOR TEST MOVING AND PLACE IN TEMP LOCATION*/

DATA TEMP.HAVE;

ARRAY VARS $200 VARS1 - VARS100;

DO J = 1 TO 1000000;

  DO I = 1 TO 100;

  VARS{I} = CATS("STUFF",CEIL(RANUNI(0)*100000000));

  END;

  OUTPUT;

END;

DROP I J;

RUN;

/*MOVE THE FILE MANUALLY TO E:\DROPZONE AND THEN EXECUTE THE FOLLOWING WHILE THE DATASET IS LOCKED TO TEST*/

%MACRO MOVEFILE (INLIB,DSNAME,OUTLIB);

%IF %SYSFUNC(EXIST(&INLIB..&DSNAME.)) %THEN %DO; /*CHECKS TO SEE IF THE DATASET EXISTS*/

   %LET DSID = %SYSFUNC(OPEN(&INLIB..&DSNAME.));  /*IF THE FILE EXISTS IT ATTEMPTS TO OPEN THE DATASET.  IF THE DATASET CAN BE OPENED THEN IT ALLOCATES A DSID */

   %IF &DSID. > 0 %THEN %DO; /*IF A DSID HAS BEEN ALLOCATED THEN CLOSES THE FILE, SO THERE IS NO LOCK ON THE FILE AND MOVES THE DATASET TO THE NEW LOCATION*/

      %LET CLOSE = %SYSFUNC(CLOSE(&DSID.));

      PROC COPY IN=&INLIB. OUT=&OUTLIB. MOVE;

        SELECT &DSNAME.;

    RUN;

   %END;

%END;

%MEND;

%MOVEFILE(DROPZONE,HAVE,MOVELOC); /*SPECIFY THE LIBRARY WHERE THE DATASET IS LOCATED, THE DATASET NAME AND THE LIBRARY THE DATASET SHOULD BE COPIED TO*/

This way the file will not be moved until there is no longer a lock on the dataset, hence reducing the chance of a file corruption.

I hope this is more beneficial.

Regards,

Scott

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 11 replies
  • 469 views
  • 8 likes
  • 5 in conversation