topic Showing Datastep Progress in SAS Programming

Showing Datastep Progress

FriedEgg — Fri, 26 Aug 2011 20:42:29 GMT

Problem:

I have a very long running process, due to inconsistent server load determining when the process will complete is difficult based on wall time alone. The process runs in batch mode on a unix server.

Possible Solution:

Here is an example of the functionality I would like to replicate (code below is perl). Put a note in the log each time a given number of records has been processed.

while (<INFILE>) {

$counter++;

if ($counter%500000 == 0) {

print "Working... $counter \n";

}

#Do Stuff;

}

print "All Done... $counter \n";

The above will read a file and each time it reaches an increment of 500,000 records will print a status msg while the program is working to let the user know the progress of the script.

This is what I want to replicate in SAS.

Should I add -logparm "write=immediate" to my config file, does this actually print immediately while still processing a datastep?

I am not currently able to test but I know in my current enviornment the following will not work:

Curent Enviornment: LOGPARM=WRITE=BUFFERED ROLLOVER=NONE OPEN=REPLACE

data _null_;

do i=1 to 10000000;

call ranuni(1234);

if mod(i,50000)=0 then putlog i=;

end;

run;

If the write=immediate option doesn't resolve the issue my next thought is to call sysexec and cat a note onto the end of the logfile (if I even can incase there is a lock on the file held by sas while using write=buffered?

Has anyone tried something like this?

Have any other ideas to implement a similar function?

Showing Datastep Progress

LinusH — Fri, 02 Sep 2011 14:39:14 GMT

Have you tried write=immediate?

You might want combine it with an altlog specification.

/Linus

Showing Datastep Progress

art297 — Sat, 03 Sep 2011 23:01:31 GMT

You also might want to post your question over on SAS-L (http://www.listserv.uga.edu/cgi-bin/wa?A0=sas-l&D=1&H=0&O=D&T=1) as Michael Raithel looks there, but not here, and he is a whiz at coming up with such solutions. Just a thought.

Showing Datastep Progress

FriedEgg — Tue, 29 Nov 2011 21:37:21 GMT

I finally had a chance to go back and test this.

I was downloading a very large dataset (10 billion rows, 500GB) across a slow WAN connection from an Oracle database I knew it would take many hours to complete (12 to be exact) and I wanted to have an additional way to make sure observations were getting written other than logging into the file system and checking file sizes and network traffic etc...

data das.segment_broadcast_fact;

if not mod(_n_,500000000) then put _n_=;

set oradb2.web_session_fact

( keep = dt

client_nbr

user_session_id

broadcast_seg_type_cd

dbsliceparm = (ALL,10)

where=(month(dt)=10)

);

run;

with this code I get a note immeditaely to my log every 500 millionth iteration in the datastep.