Read the SAS .log files and showing error message in a particular Table

[Shakti_Sourav]

Dear Team,

I am trying to read the all .log files from the server and show the error message and flow name in a table.

The scheduled flow format is in a server like FLOWNAME_YEAR_MONTH_DATE_HOUR_MINUTES_SECOND.log

I have mentioned a code for reference.

In the above code, I want to the current year, month, and day, And want to extract the scheduled_flow and Table name from another table dynamically which is in the Work library.

then after filtering the year and monthly basis of log files store it in a table. t

The table format is,

Flow Name        Table Name         Error Message

Note: In the code, Mentioned * means, its refers to the rest of time part.

Please suggest, How to do extract the log files dynamically with current date ?

extract the flowName_tableName_year_month_day_whatever_time as it is ?

Let me know, If you find any issues.

Regards,

Shakti Sourav

[Tom]

Do you mean that the file names look like:

 FLOWNAME_2023_12_23_13_01_23.log

If so why are you looking for filenames like:

XXXXXXX_2023.12.23*.log

Either way you should be able to generate a the pattern with the day of the year part easily enough.  To generate periods use the YYMMDDP10. format.

%let date=%sysfunc(date(),yymmddp10.);

To convert to underscores use TRANSLATE() function.

%let date=%sysfunc(translate(&date),_,.));

Then your data step code should be simple.  I am not sure what you want to READ from the file, so fill that part in yourself.  Keep the full filename (or at least the time of day part) in case the job ran more than one on the day of interest.

data want;
  infile "&topdir/&flow._&table._&date*.log" filename=fname .... ;
  input .... ;
  flow = "&flow";
  table = "&table";
  date = input("&date",yymmdd10.);
  format date yymmdd10.;
  filename = fname;
  ....
run;

[Shakti_Sourav]

Thanks, Tom for your quick reply.

The scheduled flow name looks like FlowName_TableName_Year_Month_Day_Hour_Minutes_Seconds in the Server.

I have mentioned one example for your clarity,

In the server, I have the same flow and table name in different Hours, Minutes, and Seconds. so Want to read the flow, tablename, year wise all the log files then extract the only error or warning messages.

Ex-

Actually on the Server,

FLOWNAME_TABLENAME_2023_12_27_08_20_02.log

FLOWNAME_TABLENAME_2023_12_27_02_30_10.log

FLOWNAME_TABLENAME_2023_12_26_08_20_02.log

FLOWNAME_TABLENAME_2023_12_26_02_30_10.log

Want, Current day month year wise all the log files. 

FLOWNAME_TABLENAME_2023_12_27_08_20_02.log

FLOWNAME_TABLENAME_2023_12_27_02_30_10.log

format is  FlowName_TableName_Year_Month_Day_Hour_Minutes_Seconds.

I want to read all the log files by FlowName, Table_name, and current day and Year and Month wise.

I have tried the * mark to read the all the time part files.

Thanks

Shakti

[Patrick]

If I understand what you wrote right then you've got already a Work table with all the log files. All you're missing is to extract info/sub-strings from these log names so you can filter on them to then use this sub-set as input to some log analyze program. If so then below self contained sample code will hopefully give you all the guidance/ideas you need.

/* create sample "log" files and dataset with all path and name of these log files */
%let path=%sysfunc(pathname(work));
data _null_;
  file "&path\FLOWNAME_TABLENAME_2023_12_27_08_20_02.log";
  put 'some log message';
  file "&path\FLOWNAME_TABLENAME_2023_12_27_02_30_10.log";
  put 'some log message'/ 'line 2';
  file "&path\FLOWNAME_TABLENAME_2023_12_26_08_20_02.log";
  put 'some log message';
  file "&path\FLOWNAME_TABLENAME_2023_12_26_02_30_10.log";
  put 'some log message';
run;

data log_list;
  infile datalines truncover;
  input log $1000.;
  log=cats("&path\",log);
  
  length 
    flowname $100
    tablename $32
    flow_dttm 8
    _flow_dttm_c $15
    ;
  format flow_dttm datetime20.;

  _basename=scan(log,-1,'\/');
  flowname=scan(_basename,1,'_');
  tablename=scan(_basename,2,'_');

  _flow_dttm_c=prxchange('s/^.+(\d\d\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d)\.log/$1$2$3T$4$5$6/',1,strip(_basename));
  flow_dttm=input(_flow_dttm_c,b8601dt15.);
  drop _:;
  datalines;
FLOWNAME_TABLENAME_2023_12_27_08_20_02.log
FLOWNAME_TABLENAME_2023_12_27_02_30_10.log
FLOWNAME_TABLENAME_2023_12_26_08_20_02.log
FLOWNAME_TABLENAME_2023_12_26_02_30_10.log
;

/* read logs and scan for words or patterns */
data log_scan;
  set log_list;
  do until(done);
    infile dummy filevar=log end=done truncover;
    input log_line $128.;
    /*** and here you scan your log ***/
    /*if find(.....) then output;*/
  end;
run;

[Shakti_Sourav]

Dear Team,

Please suggest, How to show the failed job and error or warning message in SAS EG and store it in a table by flow name and error message.

Log files are stored on the server. 

read all log files which are in FLOWNAME_TABLENAME_YEAR_MONTH_DAY_HOUR_MINUTES_SECONDS.Log

Extract the log files by flow name, table name, year, month, and day, and all the times part.log

Thanks 

Shakti

[Kurt_Bremser]

I moved this in here, where it belongs; please do not start multiple threads dealing with the same issue.