DATA Step, Macro, Functions and more

too many conditional statements for large datasets

Accepted Solution Solved
Reply
Contributor
Posts: 45
Accepted Solution

too many conditional statements for large datasets

so i have coded thousands of macro variables each containing a if statement that i need to process in a data step.

ideally, i can just make 1 macro and run it.  however, it's too big for 1 macro variable to handle.

so i have no choice but to create thousands.

 

so how do i call each of the macro variable in 1 data step?

right now, i am doing a loop which will call 1 macro at a time but it's just terrible inefficient.

 

example,

%do i=1 %to &count %by 1;

data step;

 set step;

 &macro&i;

run;

%end;

 

something like this.

how can i make this into 1 data step and run all macro variables within?

 

thanks a bunch.


Accepted Solutions
Solution
‎07-05-2017 02:56 AM
SAS Super FREQ
Posts: 682

Re: too many conditional statements for large datasets

The ineffciency comes from reading the same data for each of your if statements. You should code it like this:

%macro test;

  data step;
    set step;

    %do i=1 %to &count %by 1;
      &macro&i;
    %end;
  run;

%mend;

%test

In this way the macro will generate all the IF statements in one DATA Step and the data is only red once.

 

Bruno

View solution in original post


All Replies
Super User
Posts: 17,747

Re: too many conditional statements for large datasets

Can you change your design to use DOSUBL or %include code from a text file? 

 

Can you post a small example of what you're process is so we can make better suggestion for improvement or alternatives? 

 


Grumbler wrote:

so i have coded thousands of macro variables each containing a if statement that i need to process in a data step.

ideally, i can just make 1 macro and run it.  however, it's too big for 1 macro variable to handle.

so i have no choice but to create thousands.

 

so how do i call each of the macro variable in 1 data step?

right now, i am doing a loop which will call 1 macro at a time but it's just terrible inefficient.

 

example,

%do i=1 %to &count %by 1;

data step;

 set step;

 &macro&i;

run;

%end;

 

something like this.

how can i make this into 1 data step and run all macro variables within?

 

thanks a bunch.


 

Solution
‎07-05-2017 02:56 AM
SAS Super FREQ
Posts: 682

Re: too many conditional statements for large datasets

The ineffciency comes from reading the same data for each of your if statements. You should code it like this:

%macro test;

  data step;
    set step;

    %do i=1 %to &count %by 1;
      &macro&i;
    %end;
  run;

%mend;

%test

In this way the macro will generate all the IF statements in one DATA Step and the data is only red once.

 

Bruno

Contributor
Posts: 45

Re: too many conditional statements for large datasets

thanks everyone. problem solved. forgot about using macro loops inside the data step. Smiley Happy
Super User
Super User
Posts: 7,392

Re: too many conditional statements for large datasets

I would really be suprised if the use of thousands of macro variables is the best method to process data.  Base SAS is the executable language for processing data, and is fully functional, macro does not do anything other than generate this code.  If you can provide some examples I can show better methods to get the output.  For example there is data modelling, i.e. restructuring data to make coding more efficient - one thing I always see is the "Excel" mentality, having data going across the page which is ok for people to look at, but sub optimal to program with.  Next there is lots of functionality within the base langauge to perform various functions against arrays of variables and over numbers of rows.  

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 108 views
  • 1 like
  • 4 in conversation