Read the file names and create a dataset

[Babloo]

I've trouble understanding your suggestions since I'm a novice user. However, I tried  your code as like below, but still i could not succeed. This time there was no error either. Instead the code failed to read records from the infiles although there was a records in one of the file.

Aplogies for testing for patience. I can understand if someone can tweak my code to work in such a way as I excepted instead of writing.

Appreciate all your efforts so far.

Code:

libname log '/usr/sas/tir/test/loganalysis/';

Options symbolgen mlogic mprint mfile; 

%let path=/usr/sas/tir/test/loganalysis ;

libname log "&path/";

data log.output_file ;
  infile "&path/output_file.txt" firstobs=2 truncover ;
  input fname $400. ;
run;


%macro log_analysis;

%let filedate=%sysfunc(putn("&sysdate9"d-3,yymmdd10.));
%put &filedate;

data log.log_analysis;
set log.output_file;
filename=fname;
length fname $400;
infile dummy dsd truncover FILEVAR = fname END = end_of_file LRECL=32000;
DO WHILE (not end_of_file);
input var : $ 3000.;
/*filename=fname;*/
var1 = _infile_;
if var1 = :'201';
Date_TimeStamp= scan(var1,1," ");
Status = scan(var1,2," ");
Processid = scan(var1,3," ");
userid = scan(var1,4," ");
Details = scan(var1,-1,'-');
drop var var1;
output;
end;
run;

%mend log_analysis;

data _null_;
  set log.output_file;
  if fname =: '/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer' then call execute ('%log_analysis;');
else put 'no log files';
run;

Piece of log:

NOTE: The infile DUMMY is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-15_tmptcmsaslva2_18208.log,
       Access Permission=rwxrwxr-x,
      Last Modified=Thu Sep 17 03:57:47 2015,
      File Size (bytes)=1831

NOTE: The infile DUMMY is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-15_tmptcmsaslva2_19142.log,
      Access Permission=rwxrwxr-x,
      Last Modified=Tue Sep 15 23:57:49 2015,
      File Size (bytes)=2254685

NOTE: 1 record was read from the infile DUMMY.
      The minimum record length was 572.
      The maximum record length was 572.
NOTE: 1 record was read from the infile DUMMY.
      The minimum record length was 572.
      The maximum record length was 572.
NOTE: There were 2 observations read from the data set LOG.OUTPUT_FILE.
NOTE: The data set LOG.LOG_ANALYSIS has 0 observations and 6 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.01 seconds

Ouput_file.txt has records like,

fname
/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-15_tmptcmsaslva2_18208.log
/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-15_tmptcmsaslva2_19142.log

[Tom]

This is the subsetting IF statement that for some reason you keep adding back to the program.

if var1 = :'201';

What this means is :

if var1 = :'201' THEN RETURN;

So when your program read the first line of each file and saw that they did not start with the letters '201' it went back to the top of the data step and started reading the next file. That is why only one line was read from each file.

Instead you want to write an IF/THEN/DO loop so that you can conditionally process each line of the file without leaving the DO WHILE loop.

if var1 = :'201' THEN DO;
  .... code to execute conditionally goes here ...
END;

[Babloo]

Excellent, code works like a charm after I modified my if clause as you said, however i do have other questions on total number of observations in log.log_analysis.

I would except atleast 9+18702=18711 records in the datset log.log_analysis. Any thoughts why I got less observations when compared to the required observations?

NOTE: The infile DUMMY is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-15_tmptcmsaslva2_18208.log,
      Access Permission=rwxrwxr-x,
      Last Modified=Thu Sep 17 03:57:47 2015,
      File Size (bytes)=1831

NOTE: The infile DUMMY is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-15_tmptcmsaslva2_19142.log,
      Access Permission=rwxrwxr-x,
      Last Modified=Tue Sep 15 23:57:49 2015,
      File Size (bytes)=2254685

NOTE: 9 records were read from the infile DUMMY.
      The minimum record length was 107.
      The maximum record length was 572.
NOTE: 18702 records were read from the infile DUMMY.
      The minimum record length was 66.
      The maximum record length was 572.
NOTE: There were 2 observations read from the data set LOG.OUTPUT_FILE.
NOTE: The data set LOG.LOG_ANALYSIS has 18705 observations and 6 variables.

[Kurt_Bremser]

Make sure that the OUTPUT; statement is not inside a conditional block.

[Tom]

The number of records output will be the number of records read that start with '201'.  It could be zero , all or any number in between for each file.  If you want a different result then put different logic inside the DO WHILE loop.

[Kurt_Bremser]

When debugging these kinds of problems, start with a simple step that just reads all lines into a variable (what you already do with var) and avoid any further processing beyond the absolutely necessary output; statement. Once that works, add code until you're either finished or something unexpected happens; in this case you will have a clue what caused this.

Don't try to solve everything at once, work towards your goal in easily controllable steps; right now you are doing too much too quickly, which then overwhelms your still limited capacity (after all, you're just a beginner, nothing wrong with that) to discern where the problem might be.

[Babloo]

I've a other question now, how to code to look for the next file and create a SAS dataset when the first file is in use. When I tried reading other log files, I got error as follows.

ERROR: File is in use, /usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-11_tmptcmsaslva2_29115.log.

Output_file.txt has,

fname
/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-11_tmptcmsaslva2_29115.log
/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-11_tmptcmsaslva2_18208.log
/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-09-11_tmptcmsaslva2_19142.log 

[Tom]

Open a new question for that topic.  If you can solve the issue of testing if the file can be opened then you just pass the resulting list of readable files to the program generated in response to this question.

[Kurt_Bremser]

Follow my previous tip and solve the problem in simple, modular steps.

Once you have a simple step that processes only one file within a macro definition, you can test for the availability of that file immediately before processing it. Looping through the macro will then only omit the locked files, without causing a show-stopper.

[Babloo]

Thanks again for your quick reply. This is the last task to complete the project and there was no time  to explore this topic.

However, I will ensure to read this stuff during weekends to avoid depedency on master like you.

May I request you to give simple example for looping through the macro to omit the locked files inline with my task?

[Kurt_Bremser]

Here http://www.wuss.org/proceedings11/Papers_Galligan_O_74889.pdf you find how to use the fopen() function to detect if an external file can be opened.

[Babloo]

Thanks for the document. If I've one file in infile statement, I can achive this easily.

I wonder how to apply this logic when my filenames are in .txt file. How to SAS to go ahead to read next file  via fopen() when the file is locked?