topic Re: Reading a list of files and conditionally export the files in SAS Programming

Reading a list of files and conditionally export the files

Babloo — Mon, 31 Aug 2015 11:36:01 GMT

I would request someone to give suggestion for my questions.

1. I've a file called output_file.txt in unix and it has list of file names (e.g.) SASApp_STPServer_tmp_12345.log. Now I need to create a dataset by consolidating all the files read from output_file.txt.

Since files are coming from different systems (will have different folder structures) I can't use wildcards in infile statement like infile "/usr/sas/sas_config/Lev1/SASApp/Logs/SASApp_STPServer_tmp_*.log" dsd truncover filename=fname; However file layouts are same like below.

data log_analysis;

length fname filename $200;

infile "/usr/sas/sas_config/Lev1/SASApp/Logs/SASApp_STPServer_tmp_*.log" dsd truncover filename=fname;

input var : $ 3000.;

filename=fname;

var1 = _infile_;

if var1 = :'201';

Date_TimeStamp= scan(var1,1," ");

Status = scan(var1,2," ");

Processid = scan(var1,3," ");

userid = scan(var1,4," ");

Details = scan(var1,-1,'-');

drop var var1;

run;

This code will work only if file is from ' /usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/'. Now I would like to know, how should I tweak this code, if I want to read /usr/sas/tir/test/loganalysis/output_file.txt.

Assume output_file.txt has filenmaes like SASApp_STPServer_tmp_12345.log JavaApp_STPServer_tmp_34745.log SharepointApp_STPServer_tmp_72345.log

2. How to conditionally export the file? e.g. I need to export .csv only if dataset has greater than 0 observations.

Thanks in advance for your help.

Re: Reading a list of files and conditionally export the files

Babloo — Mon, 31 Aug 2015 12:14:17 GMT

I've the following question as well. In below why one data step works and the other not.

DATA _null_;
SET InputSet NOBS=nobs;

IF nobs = 0 then PUT "ERROR: InputSet Table is empty"; /* Doesn't work. */
RUN;

DATA _null_;
IF nobs = 0 then PUT "ERROR: InputSet Table is empty"; /* Works. */
SET InputSet NOBS=nobs;
RUN;

Re: Reading a list of files and conditionally export the files

Kurt_Bremser — Mon, 31 Aug 2015 12:41:02 GMT

When the SET statement encounters an end-of-file condition, it immediately stops processing of the data step, including all statements that appear after the SET statement in the data step.

So your IF in the first example is never executed in case no obs are present.

In the second example, the IF is executed before the SET can stop the datastep.

The constant (that's what it is, basically) nobs is set before the data step starts iterating.

Re: Reading a list of files and conditionally export the files

michelle_delaurentis_sas_com — Thu, 10 Sep 2015 17:26:01 GMT

Another way is to use an IF 0 THEN SET statement. During compilation time nobs is assigned a value but the SET statement never executes. This will save processing time especially if there are many observations in your data set. Since the SET statement will never execute order of the statements doesn't matter.

DATA _null_;

if 0 then set inputset nobs=nobs;

IF nobs = 0 then PUT "ERROR: InputSet Table is empty";

RUN;

Re: Reading a list of files and conditionally export the files

michelle_delaurentis_sas_com — Thu, 10 Sep 2015 17:29:34 GMT

You can use the filevar= option on the infile statement to read in multiple files into the same SAS data set, as illustrated in these Samples:

http://support.sas.com/kb/24712

http://support.sas.com/kb/24710

Hope that helps!

Re: Reading a list of files and conditionally export the files

Babloo — Fri, 11 Sep 2015 13:10:43 GMT

I've the code like below.

data dirlist;
  infile cards truncover;
  input file_names $300.;
  cards;
'/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-30_tmptcmlva2_19142.log'
'/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-29_tmptcmlva2_19142.log'
'/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-28_tmptcmlva2_19142.log'
;
run;

Now I need to read all the files and append it to a single file as mentioned in data step. So I wrote a code as below. However, I could not succeeed. See the log below.

There are records in my input files,but my code could not read it for some reasons. Please guide me.

DATA output_data_set;
set dirlist;
/* read the file references in variable called file_names */
INFILE  IN dsd truncover FILEVAR = file_names END = end_of_file LRECL=32000;
     DO WHILE (end_of_file = 0);
        input var : $ 3000.;
/*filename=file_names;*/
var1 = _infile_;
if var1 = :'201';
Date_TimeStamp= scan(var1,1," ");
Status = scan(var1,2," ");
Processid = scan(var1,3," ");
userid = scan(var1,4," ");
Details = scan(var1,-1,'-');
drop var var1;
        OUTPUT;
     END;
RUN;

Log:

4          data dirlist;
5            infile cards truncover;
6            input file_names $300.;
7            cards;

NOTE: The data set WORK.DIRLIST has 3 observations and 1 variables.
NOTE: DATA statement used (Total process time):
      real time           0.04 seconds
      cpu time            0.00 seconds
      

7        !         
11         ;
12         run;
2                                                          The SAS System                           08:36 Friday, September 11, 2015

13         
14         DATA output_data_set;
15         set dirlist;
16         /* read the file references in variable called file_names */
17         INFILE  IN dsd truncover FILEVAR = file_names END = end_of_file LRECL=32000;
18              DO WHILE (end_of_file = 0);
19                 input var : $ 3000.;
20         /*filename=file_names;*/
21         var1 = _infile_;
22         if var1 = :'201';
23         Date_TimeStamp= scan(var1,1," ");
24         Status = scan(var1,2," ");
25         Processid = scan(var1,3," ");
26         userid = scan(var1,4," ");
27         Details = scan(var1,-1,'-');
28         drop var var1;
29                 OUTPUT;
30              END;
31         RUN;

NOTE: The variable file_names exists on an input data set, but was also specified in an I/O statement option.  The variable will 
      not be included on any output data set.
NOTE: The infile IN is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-30_tmptcmlva2_19142.log,
      
      File List=('/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-30_tmptcmlva2_19142.log' 
      '/apps/tir/test/loganalysis/
'),
      Owner Name=sassrv,Group Name=sas,
      Access Permission=rwxrwxr-x,
      Last Modified=Mon Aug 31 00:01:56 2015,
      File Size (bytes)=720559

NOTE: The infile IN is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-29_tmptcmlva2_19142.log,
      
      File List=('/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-29_tmptcmlva2_19142.log' 
      '/apps/tir/test/loganalysis/
'),
      Owner Name=sassrv,Group Name=sas,
      Access Permission=rwxrwxr-x,
      Last Modified=Sun Aug 30 00:01:46 2015,
      File Size (bytes)=720559

NOTE: The infile IN is:
      Filename=/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-28_tmptcmlva2_19142.log,
      
      File List=('/usr/sas/sas_config/Lev1/SASApp/StoredProcessServer/Logs/SASApp_STPServer_2015-08-28_tmptcmlva2_19142.log' 
      '/apps/tir/test/loganalysis/
'),
      Owner Name=sassrv,Group Name=sas,
      Access Permission=rwxrwxr-x,
      Last Modified=Sat Aug 29 00:01:36 2015,
      File Size (bytes)=2815408

NOTE: 1 record was read from the infile IN.
      The minimum record length was 572.
      The maximum record length was 572.
NOTE: 1 record was read from the infile IN.
      The minimum record length was 572.
      The maximum record length was 572.
3                                                          The SAS System                           08:36 Friday, September 11, 2015

NOTE: 1 record was read from the infile IN.
      The minimum record length was 572.
      The maximum record length was 572.
NOTE: There were 3 observations read from the data set WORK.DIRLIST.
NOTE: The data set WORK.OUTPUT_DATA_SET has 0 observations and 5 variables.
NOTE: DATA statement used (Total process time):
      real time           0.07 seconds
      cpu time            0.00 seconds

Re: Reading a list of files and conditionally export the files

RamKumar — Mon, 14 Sep 2015 06:34:56 GMT

I'm looking for some guidance to complete this task. Thanks in advance.