topic Re: Parse SAS Log file in SAS Programming

Parse SAS Log file

Saurabh13 — Tue, 09 May 2017 07:21:06 GMT

I have a sas log . For example :

NOTE: The data set WORK.A has 1 observations and 1 variables.
NOTE: DATA statement used (Total process time):
real time 0.01 seconds
user cpu time 0.01 seconds
system cpu time 0.01 seconds
memory 764.06k
OS Memory 18596.00k
Timestamp 05/08/2017 11:26:34 PM
Step Count 6 Switch Count 30
Page Faults 0
Page Reclaims 351
Page Swaps 0
Voluntary Context Switches 73
Involuntary Context Switches 1
Block Input Operations 0
Block Output Operations 312

28
29 data B;
30 set A;
31 x=3;
32 run;

NOTE: There were 1 observations read from the data set WORK.A.
NOTE: The data set WORK.B has 1 observations and 2 variables.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
user cpu time 0.01 seconds
2 The SAS System 23:11 Monday, May 8, 2017

system cpu time 0.00 seconds
memory 1032.62k
OS Memory 18856.00k
Timestamp 05/08/2017 11:26:34 PM
Step Count 7 Switch Count 40
Page Faults 0
Page Reclaims 297
Page Swaps 0
Voluntary Context Switches 111
Involuntary Context Switches 0
Block Input Operations 0
Block Output Operations 328

33
34 proc sql;
35 create table XYZ as select * from B;
NOTE: Table WORK.XYZ created, with 1 rows and 2 columns.

36 quit;
NOTE: PROCEDURE SQL used (Total process time):
real time 0.00 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds

After Each data step or Proc step there is statistics of real time and CPU time. I want to parse and create dataset which will have dataset name and real time . Like

Table Time records

Work.A 0.01 1

Work.B 0.00 1

Work.XYZ 0.00 1

I am trying like this but not able to capture Table /dataset name .

filename in "log_path";

data x;
infile inn missover truncover;
input text $1000. ;
b=strip(text);
x=compress(text);
c=substr(x,1,8);
zz=compress(x,'realtime');
if text eq '' then delete;
if c='realtime';
run;

could someone please help ?

Thanks,

Re: Parse SAS Log file

Kurt_Bremser — Tue, 09 May 2017 07:36:52 GMT

Please repost your log snippet using the {i} icon, so that the original formatting is preserved.

Re: Parse SAS Log file

Saurabh13 — Tue, 09 May 2017 07:46:54 GMT

25         data A;
26          s=2;
27         run;

NOTE: The data set WORK.A has 1 observations and 1 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      user cpu time       0.01 seconds
      system cpu time     0.00 seconds
      memory              765.00k
      OS Memory           18340.00k
      Timestamp           05/09/2017 12:41:45 AM
      Step Count                        2  Switch Count  29
      Page Faults                       0
      Page Reclaims                     425
      Page Swaps                        0
      Voluntary Context Switches        67
      Involuntary Context Switches      0
      Block Input Operations            0
      Block Output Operations           304
      

28         
29         data B;
30          set A;
31          x=3;
32         run;

NOTE: There were 1 observations read from the data set WORK.A.
NOTE: The data set WORK.B has 1 observations and 2 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      user cpu time       0.00 seconds
2                                                          The SAS System                                 00:41 Tuesday, May 9, 2017

      system cpu time     0.00 seconds
      memory              1030.53k
      OS Memory           18600.00k
      Timestamp           05/09/2017 12:41:45 AM
      Step Count                        3  Switch Count  39
      Page Faults                       0
      Page Reclaims                     314
      Page Swaps                        0
      Voluntary Context Switches        92
      Involuntary Context Switches      0
      Block Input Operations            0
      Block Output Operations           320
      

33         
34         proc sql;
35          create table XYZ as select * from B;
NOTE: Table WORK.XYZ created, with 1 rows and 2 columns.

36         quit;
NOTE: PROCEDURE SQL used (Total process time):
      real time           0.00 seconds
      user cpu time       0.01 seconds
      system cpu time     0.00 seconds
      memory              5419.12k
      OS Memory           23464.00k
      Timestamp           05/09/2017 12:41:45 AM
      Step Count                        4  Switch Count  29
      Page Faults                       0
      Page Reclaims                     149
      Page Swaps                        0
      Voluntary Context Switches        67
      Involuntary Context Switches      0
      Block Input Operations            0
      Block Output Operations           304

Re: Parse SAS Log file

Shmuel — Tue, 09 May 2017 07:50:11 GMT

Each log line has its own format.

Use functions index or find to check for wanted keywords like DATA, OUT=, observations, real time, etc.

When found use function scan to get the desired information (dataset name or number of observations or measured time).

In your code, you used function compress. Change it to function compbl or even skip it.

Use function scan instead function substr.

Try it and come with your new code and results.

Re: Parse SAS Log file

Kurt_Bremser — Tue, 09 May 2017 08:19:32 GMT

Concur with @Shmuel. use the index() and scan() functions; index() to detect the presence of a certain string, and scan() to extract a certain "word" from the string.

The data step would look like that:

- declare the infile with truncover and a end=done option

- retain variables for datasetname, number of records and real time

- set length for datasetname to $32

- when you encounter a index(text,"NOTE: The data set") = 1, do:

- if datasetname is already set (indicating that you've been collecting data already), output

- set datasetname with scan(text,5," ") and extract nrecs with nrecs = input(scan(text,7," ",best.)

- set realtime to missing

when you encounter "real time" in text, extract realtime like nrecs (experiment until you get the correct "word" number)

when if done is true and datasetname > " ", output (to write the last occurence)

The check for the NOTEs from SQL look similar to that for a data step, just the words and positions are different

Re: Parse SAS Log file

Saurabh13 — Tue, 09 May 2017 23:22:09 GMT

Tried but in 2 steps.

data x_wo_step(keep= text ds_name ds_count real_time ) w_step(keep=ds_name step_used);
infile inn truncover end=eof;
input text $1000. ;
retain ds_name ds_count ;
if indexw(text,"NOTE: The data set") eq 1 then do;
ds_name = scan(text,5," ");
ds_count = scan(text,7," ");
output x_wo_step;
end;
else if indexw(text,"NOTE: Table") eq 1 then do;
ds_name = scan(text,3," ");
ds_count = scan(text,6," ");
output x_wo_step;
end;
else if indexw(text,'real') eq 1 then do; 
real_time = scan(text,3," "); 
output x_wo_step;
end;
else if indexw(text,'used') gt 1 then do;
step_used = scan(text,2," ")||""|| scan(text,3," "); /*--It's data/proc step --*/
output w_step;
end;
else do;
delete;
end;
run;
data x_wo_step_1;
set x_wo_step;
if real_time eq '' then delete;
run;
data Final_stats(drop=text);
merge x_wo_step_1(in=a) w_step(in=b);
by ds_name;
run;

Re: Parse SAS Log file

Kurt_Bremser — Wed, 10 May 2017 05:46:29 GMT

My take would be this:

data log (keep=datasetname nrecs realtime);
infile in truncover end=eof;
retain datasetname nrecs realtime;
format
  datasetname $32.
  nrecs comma12.
  realtime time11.2
;
input text $100.;
if index(text,"NOTE: The data set") = 1
then do;
  if datasetname ne " " then output;
  datasetname = scan(text,5," ");
  nrecs = input(scan(text,7," "),best.);
  realtime = .;
end;
else if index(text,"NOTE: Table") = 1
then do;
  if datasetname ne " " then output;
  datasetname = scan(text,3," ");
  nrecs = input(scan(text,6," "),best.);
  realtime = .;
end;
if index(text,"real time") > 0 then realtime = input(scan(text,3," "),best.);
if eof and datasetname ne " " then output;
run;

Since PROC SQL issues a "Note: Table" for every dataset created, but only writes the statistics when it quits, one must not rely on "real time" for any output.

Also note that your approach would create wrong data when the same dataset is created in several steps within one log file.