PIPE Command Takes Too Long?

[shl007]

Hello - my program using a PIPE command is running a long time. Hanging, maybe. Below is the code. I'm using SAS Enterprise Guide 7.15. Any tips on what might be causing my code to hang, or how I might improve performance?

The multiple "else if" statements have me wondering if that's the culprit ... I'm using around 30 of "else if" statements to handle varying requirements by dataset (each dataset can have different attributes, and there is no pattern to the data that I could automate via a macro or array).  Thanks!

filename output pipe 'ls -a /my path /.*csv'; 

 

data prototype2;length fil2read $1000;

 

   INFILE output TRUNCOVER ;/*Read pipe in again. */

   INPUT name $1000.;/*Pull pipe in*/

   fil2read=name;

   INFILE dummy FILEVAR=fil2read DSD END=last missover FIRSTOBS=9 LENGTH=len ;

    

     generate_date = input(substr(name,44,8),yymmdd8.) ;

 

     format generate_date MMDDYY.;

 

     ct = 0;

 

if find(fil2read, 'yada yada233', 'i') ge 1 then

do;

maxdt = &accbill ;  

datasetnm = 'dataset12’;

end;

else if find(fil2read, ' yada yada455 ', 'i') ge 1 then

do;

maxdt = &adjwrite ;  

datasetnm = ‘dataset14’;

end;

 

/* etc. – I have a bunch of “else if” statements, maybe like 30 of them – to handle different dataset requirements – do you think these “else if” statements could be hanging things up? */

 

     do while(not last);

           if len > 9 and generate_date > maxdt then do;

                input;

                ct +1;

           end;

     end;

     if last then do;

        

   output;

           ct = 0;

     end;

    

 

run;

[ballardw]

I would suggest reading in just the data from the pipe into a data set.

Then use a separate date step to do the rest of the processing you are attempting. Since there isn't a second INPUT statement I kind of wonder why the second infile statement.

[Reeza]

Notice the colour change halfway through?

I don't know if this due to copy/paste but you have different brackets.

datasetnm = 'dataset12’;

I wonder if that's causing issues or just an issue with posting to here.

---

@shl007 wrote:

Hello - my program using a PIPE command is running a long time. Hanging, maybe. Below is the code. I'm using SAS Enterprise Guide 7.15. Any tips on what might be causing my code to hang, or how I might improve performance?

The multiple "else if" statements have me wondering if that's the culprit ... I'm using around 30 of "else if" statements to handle varying requirements by dataset (each dataset can have different attributes, and there is no pattern to the data that I could automate via a macro or array).  Thanks!

filename output pipe 'ls -a /my path /.*csv'; 

 

data prototype2;length fil2read $1000;

 

   INFILE output TRUNCOVER ;/*Read pipe in again. */

   INPUT name $1000.;/*Pull pipe in*/

   fil2read=name;

   INFILE dummy FILEVAR=fil2read DSD END=last missover FIRSTOBS=9 LENGTH=len ;

    

     generate_date = input(substr(name,44,8),yymmdd8.) ;

 

     format generate_date MMDDYY.;

 

     ct = 0;

 

if find(fil2read, 'yada yada233', 'i') ge 1 then

do;

maxdt = &accbill ;  

datasetnm = 'dataset12’;

end;

else if find(fil2read, ' yada yada455 ', 'i') ge 1 then

do;

maxdt = &adjwrite ;  

datasetnm = ‘dataset14’;

end;

 

/* etc. – I have a bunch of “else if” statements, maybe like 30 of them – to handle different dataset requirements – do you think these “else if” statements could be hanging things up? */

 

     do while(not last);

           if len > 9 and generate_date > maxdt then do;

                input;

                ct +1;

           end;

     end;

     if last then do;

        

   output;

           ct = 0;

     end;

    

 

run;

---

[Patrick]

@shl007

Just a few observations:

1. In the code you've posted below line used two different kinds of quotes. That should return a syntax error. Change the closing quote to the type of the opening quote.

datasetnm = 'dataset12’;

2. For cases where LEN is <=9  you'll end up with an infinite loop in below logic as you only issue an INPUT statement when the condition is TRUE. So once you have a NOT TRUE you'll never progress to the next record and though you'll never be DONE.

    if len > 9 and generate_date > maxdt then
      do;
        input;
        ct +1;
      end;

You need to always issue an INPUT statement to iterate through the source records. You then test what you've read and branch in your logic accordingly.