DATA Step, Macro, Functions and more

Showing Datastep Progress

Accepted Solution Solved
Reply
Trusted Advisor
Posts: 1,301
Accepted Solution

Showing Datastep Progress

Problem:

I have a very long running process, due to inconsistent server load determining when the process will complete is difficult based on wall time alone.  The process runs in batch mode on a unix server.

Possible Solution:

Here is an example of the functionality I would like to replicate (code below is perl).  Put a note in the log each time a given number of records has been processed.

while (<INFILE>) {

  $counter++;

  if ($counter%500000 == 0) {

   print "Working... $counter \n";

  }

#Do Stuff;

}

print "All Done... $counter \n";

The above will read a file and each time it reaches an increment of 500,000 records will print a status msg while the program is working to let the user know the progress of the script.

This is what I want to replicate in SAS.

Should I add -logparm "write=immediate" to my config file, does this actually print immediately while still processing a datastep?

I am not currently able to test but I know in my current enviornment the following will not work:

Curent Enviornment: LOGPARM=WRITE=BUFFERED ROLLOVER=NONE OPEN=REPLACE

data _null_;

  do i=1 to 10000000;

   call ranuni(1234);

   if mod(i,50000)=0 then putlog i=;

end;

run;

If the write=immediate option doesn't resolve the issue my next thought is to call sysexec and cat a note onto the end of the logfile (if I even can incase there is a lock on the file held by sas while using write=buffered?

Has anyone tried something like this?

Have any other ideas to implement a similar function?


Accepted Solutions
Solution
‎11-29-2011 04:37 PM
Trusted Advisor
Posts: 1,301

Showing Datastep Progress

I finally had a chance to go back and test this.

I was downloading a very large dataset (10 billion rows, 500GB) across a slow WAN connection from an Oracle database Smiley Sad  I knew it would take many hours to complete (12 to be exact) and I wanted to have an additional way to make sure observations were getting written other than logging into the file system and checking file sizes and network traffic etc...

data das.segment_broadcast_fact;

  if not mod(_n_,500000000) then put _n_=;

set oradb2.web_session_fact

  ( keep = dt

           client_nbr

           user_session_id

           broadcast_seg_type_cd

    dbsliceparm = (ALL,10)

    where=(month(dt)=10)

  );

run;

with this code I get a note immeditaely to my log every 500 millionth iteration in the datastep. Smiley Happy

View solution in original post


All Replies
Super User
Posts: 5,424

Showing Datastep Progress

Have you tried write=immediate?

You might want combine it with an altlog specification.

/Linus

Data never sleeps
PROC Star
Posts: 7,467

Showing Datastep Progress

You also might want to post your question over on SAS-L (http://www.listserv.uga.edu/cgi-bin/wa?A0=sas-l&D=1&H=0&O=D&T=1) as Michael Raithel looks there, but not here, and he is a whiz at coming up with such solutions.  Just a thought.

Solution
‎11-29-2011 04:37 PM
Trusted Advisor
Posts: 1,301

Showing Datastep Progress

I finally had a chance to go back and test this.

I was downloading a very large dataset (10 billion rows, 500GB) across a slow WAN connection from an Oracle database Smiley Sad  I knew it would take many hours to complete (12 to be exact) and I wanted to have an additional way to make sure observations were getting written other than logging into the file system and checking file sizes and network traffic etc...

data das.segment_broadcast_fact;

  if not mod(_n_,500000000) then put _n_=;

set oradb2.web_session_fact

  ( keep = dt

           client_nbr

           user_session_id

           broadcast_seg_type_cd

    dbsliceparm = (ALL,10)

    where=(month(dt)=10)

  );

run;

with this code I get a note immeditaely to my log every 500 millionth iteration in the datastep. Smiley Happy

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 3 replies
  • 411 views
  • 6 likes
  • 3 in conversation