SAS runs slower with each macro call simulated power

[n6b]

I'm running simulations for power & sample size.  I generate the random data, separate it into two datasets (call them x and y), then I use dataset x in a proc freq and output to a new dataset x2, then I use dataset y in a proc freq and output to a new dataset y2.

Doing that is one set of 10,000 simulations and it gives me one power estimate.  But this is one of those cases where you give it an N, it gives you the power, then you have to adjust the N and run again and it gives you power again, etc.  And you keep doing it until you hone in on the n that gives you 80% power.  And I have to do this general thing for many different cases.

So I open up SAS and submit my first macro call to estimate power and it runs pretty quickly.  Then I submit another and it runs more slowly.  Then another and it runs more slowly yet.  Etc.  And in looking at the log it is the proc freq that it is taking the time.  I do some other things but they're all fast.  The first time through the proc freq takes about a minute.  The next time maybe 2.5.  The next time maybe 4.5.  Etc.

As you can see, this is not feasible when you have a lot of results to generate.  One way around it is to open SAS, run it once, get the result, shut down SAS and re-open it, then run the next one, etc.  But of course, that's a pain.  Is there some way to wipe the memory, or do whatever needs to be done, without closing SAS every time?  I have tried proc datases delete=work._all_; but that didn't make any difference.

The proc freq in question does have an "exact binomial" option in it but I tried running it with that commented out and the same phenomenon of it getting slower and slower each time still occurred.

Any help is appreciated from the All Knowledgeable Board.  Thanks in advance.

[ballardw]

Are you sure that what you are attempting to get power is not available in Proc Power? Or maybe Proc GLMPower? These procedures have options that let you set multiple things like multiple power and generate needed size N or vice versa.

Are you sending Proc Freq output to the default display? As that gets more output then just doing the display refresh may be part of the issue. I would suggest turning off the display using the Proc Freq NOPRINT option and generate data sets from Proc Freq with the desired output.

It may help to post the code of the macro so we don't have to ask a bunch of "do you have this option enabled/disabled" questions.

[n6b]

I'm trying to get power for Sensitivity and Specificity.  But I don't want to compute it one time for Sensitivity and once again for Specificity.  Rather I'm trying to get the power to say simultaneously, (a) Sensitivity > p1 and (b) Specificity > p2.  I can't just (a) find power for Sensitivity by itself and (b) find power for Specificity by itself and (c) just multiply the two together because they are not independent.  That said, I just now multiplied the power for Sensitivity in my simulations by the power for Specificity in my simulations and so far it is always close to but slightly larger than my simulated power for both Sensitivity and Specificity simultaneously.

If there was a way for proc power to give me the n for 80% power for a specified sensitivity and specificity at the same time I would be ecstatic but I don't think that's the case.

I had already turned off the proc freq display and it still was slow.

The code is below.  The Gold Standard outcome is Positive 55% of the time and Negative 45%, so the value of the macro argument p is 0.55 and the macro call at the end.  From the macro call at the end you can also see that sensitivity and specificity are both 85% and the N for this macro call is 470.  Of course all that is just for one macro call.  My ultimate goal is to find the power for various combinations of specifiied Sensitivity and Specificity, with the alternative hypothesis always being that both is greater than 77.5%.

As I said, if there's a slick way for proc power to do this all at once then I'm all for it but as far as I can tell the only way to do this is via simulations.

Here is the code.

%MACRO OBS (p,    sens,    spec,    n);
data x;
seed = 23867;

do sim = 1 to 10000;  * THIS IS THE NUMBER OF SIMULATIONS ;
      do i = 1 to &N;        * THIS IS OUR POSITED SAMPLE SIZE ;

         z = ranuni (seed);
         if 0 <= z < &P then do;
         Gold_Standard = '0_POS';

         zz = ranuni (seed);
         if 0 <= zz < &SENS then Our_Test = '0_POS';
         if &SENS <= zz <= 1 then Our_Test = '1_NEG';
      end;

      if &P <= z <= 1 then do;
         Gold_Standard = '1_NEG';
         zz = ranuni (seed);
         if 0 <= zz < &SPEC then Our_Test = '1_NEG';
         if &SPEC <= zz <= 1 then Our_Test = '0_POS';
      end;
      output;

   end;
end;
run;

data Sensitivity;    set x;    where Gold_Standard = '0_POS';   run;

proc sort data=Sensitivity;    by sim;    run;
ods select none;
ods output BinomialTest = Sensitivity2;
proc freq data=Sensitivity;
   by sim;
   tables Our_Test / binomial (p=0.775);
   exact binomial;
run;
ods select all;

********************************************************************************;
* --EACH SIMULATION WILL HAVE EITHER XPR_BIN OR XPL_BIN. WE KEEP THOSE HERE. ;
********************************************************************************;
data Sensitivity3;
   set Sensitivity2;
   if Name1 in ('XPR_BIN','XPL_BIN');
   p_sens = nValue1;
run;

*******************************************************************************************
* --WE ADD 0.50 TO THE LEFT-SIDED P-VALUES WHERE. THE IDEA IS THAT ALL OUR P-VALUES
* WILL CORRESPOND TO A RIGHT-SIDED TEST. IOW, WE ARE TESTING Ha: Sensitivity > 0.775.
*******************************************************************************************
data Sensitivity4;
   set Sensitivity3;
   if Name1 = 'XPL_BIN' then p_sens = min (1, p_sens + 0.50);
run;

*****************************************************************;
****************** SENSITIVITY ENDS ABOVE *******************;
****************** SPECIFICITY STARTS BELOW *******************;
*****************************************************************;

data Specificity;    set x;    where Gold_Standard = '1_NEG';    run;

proc sort data=Specificity;    by sim;    run;
ods select none;
ods output BinomialTest = Specificity2;
proc freq data=Specificity;
   by sim;
   tables Our_Test / binomial (p=0.225);
   exact binomial;
run;
ods select all;

********************************************************************************;
* --EACH SIMULATION WILL HAVE EITHER XPR_BIN OR XPL_BIN. WE KEEP THOSE HERE. ;
********************************************************************************;
data Specificity3;
   set Specificity2;
   if Name1 in ('XPR_BIN','XPL_BIN');
   p_spec = nValue1;
run; 

*******************************************************************************************
* --WE ADD 0.50 TO THE LEFT-SIDED P-VALUES WHERE. THE IDEA IS THAT ALL OUR P-VALUES
* WILL CORRESPOND TO A RIGHT-SIDED TEST. IOW, WE ARE TESTING Ha: Sensitivity > 0.775.
*******************************************************************************************
data Specificity4;
   set Specificity3;
   if Name1 = 'XPR_BIN' then p_spec = min (1, p_spec + 0.50);
run;

****** WE COMBINE THE RESULTS FOR SENSITIVITY AND SPECIFICITY HERE.   ******;

proc sort data=Sensitivity4;    by sim;    run;
proc sort data=Specificity4;    by sim;    run;
data Sens_Spec;
   merge Sensitivity4
              Specificity4;
              by sim;
             if p_sens < 0.05 then Sens_Good = 1;
             if p_sens >= 0.05 then Sens_Good = 0;

           if p_spec < 0.05 then Spec_Good = 1;
           if p_spec >= 0.05 then Spec_Good = 0;
run;

proc freq data=Sens_Spec;
   tables Sens_Good * Spec_Good / out=test norow nocol;
run;
proc print data=test; run;

%MEND OBS;

%OBS (0.55,   0.85,   0.85,   470);  

[n6b]

I want to cry.  I wrote a long reply to this and now I don't see it.  Okay, let me try again.  Maybe it timed out.  Okay, this time let me just copy the code and then describe what I'm doing in a separate post.

%MACRO OBS (p,    sens,    spec,  n);
data x;
   seed = 23867;

   do sim = 1 to 10000;    * THIS IS THE NUMBER OF SIMULATIONS ;
      do i = 1 to &N;             * THIS IS OUR POSITED SAMPLE SIZE ;   

         z = ranuni (seed);

         if 0 <= z < &P then do;
         Gold_Standard = '0_POS';
         zz = ranuni (seed);
         if 0 <= zz < &SENS then Our_Test = '0_POS';
         if &SENS <= zz <= 1 then Our_Test = '1_NEG';
      end;

      if &P <= z <= 1 then do;
         Gold_Standard = '1_NEG';
         zz = ranuni (seed);
         if 0 <= zz < &SPEC then Our_Test = '1_NEG';
         if &SPEC <= zz <= 1 then Our_Test = '0_POS';
      end;
      output;
   end;
end;
run;

data Sensitivity;    set x;    where Gold_Standard = '0_POS';    run;

proc sort data=Sensitivity; by sim; run;
ods select none;
ods output BinomialTest = Sensitivity2;
proc freq data=Sensitivity;
   by sim;
   tables Our_Test / binomial (p=0.775);
   exact binomial;
run;
ods select all;

********************************************************************************;
* --EACH SIMULATION WILL HAVE EITHER XPR_BIN OR XPL_BIN. WE KEEP THOSE HERE. ;
********************************************************************************;
data Sensitivity3;
   set Sensitivity2;
   if Name1 in ('XPR_BIN','XPL_BIN');
   p_sens = nValue1;
run;

*******************************************************************************************
* --WE ADD 0.50 TO THE LEFT-SIDED P-VALUES WHERE. THE IDEA IS THAT ALL OUR P-VALUES
* WILL CORRESPOND TO A RIGHT-SIDED TEST. IOW, WE ARE TESTING Ha: Sensitivity > 0.775.
*******************************************************************************************
data Sensitivity4;
   set Sensitivity3;
   if Name1 = 'XPL_BIN' then p_sens = min (1, p_sens + 0.50);
run; 

*****************************************************************;
****************** SENSITIVITY ENDS ABOVE *******************;
****************** SPECIFICITY STARTS BELOW *******************;
*****************************************************************; 

data Specificity;    set x;    where Gold_Standard = '1_NEG';    run;

proc sort data=Specificity; by sim; run;
ods select none;
ods output BinomialTest = Specificity2;
proc freq data=Specificity;
   by sim;
   tables Our_Test / binomial (p=0.225);
   exact binomial;
run;
ods select all;

********************************************************************************;
* --EACH SIMULATION WILL HAVE EITHER XPR_BIN OR XPL_BIN. WE KEEP THOSE HERE. ;
********************************************************************************;
data Specificity3;
   set Specificity2;
   if Name1 in ('XPR_BIN','XPL_BIN');
   p_spec = nValue1;
run;  

*******************************************************************************************
* --WE ADD 0.50 TO THE LEFT-SIDED P-VALUES WHERE. THE IDEA IS THAT ALL OUR P-VALUES
* WILL CORRESPOND TO A RIGHT-SIDED TEST. IOW, WE ARE TESTING Ha: Sensitivity > 0.775.
*******************************************************************************************
data Specificity4;
   set Specificity3;
   if Name1 = 'XPR_BIN' then p_spec = min (1, p_spec + 0.50);
run;

******************  COMBINE FOR RESULTS FOR SENSITIVITY AND SPECIFICITY HERE  ********;

proc sort data=Sensitivity4;    by sim;    run;
proc sort data=Specificity4;    by sim;    run;
data Sens_Spec;
    merge Sensitivity4
                Specificity4;
    by sim;
    if p_sens < 0.05 then Sens_Good = 1;
    if p_sens >= 0.05 then Sens_Good = 0;

    if p_spec < 0.05 then Spec_Good = 1;
    if p_spec >= 0.05 then Spec_Good = 0;
run;

proc freq data=Sens_Spec;
    tables Sens_Good * Spec_Good / out=test norow nocol;
run;
proc print data=test;    run;

%MEND OBS;

%OBS (0.55,    0.85,    0.85,    470); 

[n6b]

Good, it posted, now let me explain.  I want to find power for Sensitivity and Specificity simultaneously.  I could just find the power for each separately and then multiply them together, which would be easy with proc power, except they are not independent.  I want to find the power for BOTH to be above 77.5% (the p in our Ha).  That said, in the simulations I've done so far I've multiplied the power for the two and in every case it was close to (and a little more than) the power for both simultaneously.

If proc power has a way to compute power for a specified Sensitivity and Specificity at the same time I'd be ecstatic but I don't think that's the case.

I had already suppressed output for proc freq and it was still slow.

Here's what we have in a nutshell.  We have a Gold Standard that we think will be Positive 55% of the time and Negative the other 45%.  That is the first macro argument (0.55).  The next two macro arguments are the Sensitivity and Specificity that we are specifying.  In the macro call both are 0.85, although my larger goal is to find power for many combinations of these two.  And the last macro argument is the 470, which is the sample size.  We want to get 77.5% power.  So as of now, I run the macro, see the result, adjust the N appropriately and run it again, etc.  As you can imagine this will take awhile when it takes longer and longer with every macro call, thus my question.

[data_null__]

You have some comment statements with missing semicolons causing error.    After fixing that I get the result below.  Which parameter(s) do you change for the second run of the macro based on the result here?

%OBS (0.55, 0.85, 0.85, 470);  

       Sens_    Spec_                     
 Obs     Good     Good    COUNT    PERCENT 
                                           
  1       0        0        126      1.26  
  2       0        1        669      6.69  
  3       1        0       1297     12.97  
  4       1        1       7908     79.08  

[n6b]

I copied and pasted the code and tried to neaten it up for readability so that explains those issues.  Sorry for the inconvenience.  

But anyway, yes, I got exactly what you got.  For me at least, if I look at the log, the 2nd proc freq (for specificity) takes considerably longer than the 1st proc freq.  Then if I submit the macro again, the proc freq for sensitivity is longer yet.  Then the proc freq for specificity even longer yet.  And so all for each macro call.  Each one way proc freq that computes the Binomial test takes longer than the previous.  (The two-way proc freq at the very end of the macro that gives the final results always runs quickly.)

[n6b]

Oops, I didn't read the last sentence of your message.  I want to get 77.5% power.  The last line in that proc print indicates that for n=470 and with 85% Sensitivity and Specificity, we get 79.08% power.  So what I'd do is reduce the N and run it again to try to get closer to 77.5%.  And continue that process until I get to 77.5% power.

Then when I achieve that, I'll start the whole process anew but this time I'll assume 85% Sensitivity and 86% Specificity

And I'll do that for all combinations of Sensitivity and Specificity for 85 to 90%, which means a total of 6 x 6 = 36 N numbers to achieve 77.5% power for the specific combination of Sensitivity and Specificity.

[data_null__]

I think you need to show some of the actual LOG output with OPTIONS FULLSTIMER; 

MPRINT(OBS):   ods select none;
MPRINT(OBS):   ods output BinomialTest = Sensitivity2;
MPRINT(OBS):   proc freq data=Sensitivity;
MPRINT(OBS):   by sim;
MPRINT(OBS):   tables Our_Test / binomial (p=0.775);
MPRINT(OBS):   exact binomial;
MPRINT(OBS):   run;

NOTE: The data set WORK.SENSITIVITY2 has 80000 observations and 6 variables.
NOTE: There were 5333602 observations read from the data set WORK.SENSITIVITY.
NOTE: PROCEDURE FREQ used (Total process time):
      real time           1.65 seconds
      user cpu time       1.50 seconds
      system cpu time     0.15 seconds
      memory              1024.21k
      OS Memory           13148.00k
      Timestamp           03/17/2026 10:06:59 AM
      Step Count                        20  Switch Count  0

MPRINT(OBS):   ods select all;

[n6b]

Okay, I'm running it right now with "options fullstimer;"  I've run it several times in the current SAS session so by this time it may take ten minutes for these proc freqs.

When it finally runs, is a better way of getting it into this forum other than just copying and pasting it?  I mean, I can copy and paste it no problem but I want to make sure it's readable for you.

[data_null__]

This menu has several options


Here to attach files.

[n6b]

Here is the log with OPTIONS FULLSTIMER;

143 options fullstimer;
144 %OBS (0.55, 0.85, 0.85, 450); * 92.05 85.77 79.08 ;

NOTE: The data set WORK.X has 4500000 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.31 seconds
user cpu time 0.26 seconds
system cpu time 0.04 seconds
memory 410.68k
OS Memory 16368.00k
Timestamp 17/03/2026 02:02:59 PM
Step Count 80 Switch Count 0

NOTE: There were 2473632 observations read from the data set WORK.X.
WHERE Gold_Standard='0_POS';
NOTE: The data set WORK.SENSITIVITY has 2473632 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.36 seconds
user cpu time 0.34 seconds
system cpu time 0.04 seconds
memory 640.37k
OS Memory 16368.00k
Timestamp 17/03/2026 02:03:00 PM
Step Count 81 Switch Count 0

NOTE: There were 2473632 observations read from the data set WORK.SENSITIVITY.
NOTE: The data set WORK.SENSITIVITY has 2473632 observations and 7 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 0.72 seconds
user cpu time 0.20 seconds
system cpu time 0.42 seconds
memory 237094.53k
OS Memory 251600.00k
Timestamp 17/03/2026 02:03:01 PM
Step Count 82 Switch Count 0

NOTE: The data set WORK.SENSITIVITY2 has 80000 observations and 6 variables.
NOTE: There were 2473632 observations read from the data set WORK.SENSITIVITY.
NOTE: PROCEDURE FREQ used (Total process time):
real time 8:16.47
user cpu time 52.11 seconds
system cpu time 7:21.62
memory 1043.09k
OS Memory 16368.00k
Timestamp 17/03/2026 02:11:17 PM
Step Count 83 Switch Count 2

NOTE: There were 80000 observations read from the data set WORK.SENSITIVITY2.
NOTE: The data set WORK.SENSITIVITY3 has 10000 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
user cpu time 0.01 seconds
system cpu time 0.00 seconds
memory 630.81k
OS Memory 16368.00k
Timestamp 17/03/2026 02:11:17 PM
Step Count 84 Switch Count 0

NOTE: There were 10000 observations read from the data set WORK.SENSITIVITY3.
NOTE: The data set WORK.SENSITIVITY4 has 10000 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.01 seconds
user cpu time 0.00 seconds
system cpu time 0.01 seconds
memory 634.18k
OS Memory 16368.00k
Timestamp 17/03/2026 02:11:17 PM
Step Count 85 Switch Count 0

NOTE: There were 2026368 observations read from the data set WORK.X.
WHERE Gold_Standard='1_NEG';
NOTE: The data set WORK.SPECIFICITY has 2026368 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.35 seconds
user cpu time 0.14 seconds
system cpu time 0.20 seconds
memory 640.37k
OS Memory 16368.00k
Timestamp 17/03/2026 02:11:18 PM
Step Count 86 Switch Count 0

NOTE: There were 2026368 observations read from the data set WORK.SPECIFICITY.
NOTE: The data set WORK.SPECIFICITY has 2026368 observations and 7 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 0.26 seconds
user cpu time 0.31 seconds
system cpu time 0.17 seconds
memory 195177.28k
OS Memory 210240.00k
Timestamp 17/03/2026 02:11:18 PM
Step Count 87 Switch Count 0

NOTE: The data set WORK.SPECIFICITY2 has 80000 observations and 6 variables.
NOTE: There were 2026368 observations read from the data set WORK.SPECIFICITY.
NOTE: PROCEDURE FREQ used (Total process time):
real time 9:37.08
user cpu time 57.09 seconds
system cpu time 8:36.96
memory 1063.34k
OS Memory 16368.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 88 Switch Count 2

NOTE: There were 80000 observations read from the data set WORK.SPECIFICITY2.
NOTE: The data set WORK.SPECIFICITY3 has 10000 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.01 seconds
user cpu time 0.00 seconds
system cpu time 0.01 seconds
memory 630.81k
OS Memory 16368.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 89 Switch Count 0

NOTE: There were 10000 observations read from the data set WORK.SPECIFICITY3.
NOTE: The data set WORK.SPECIFICITY4 has 10000 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
user cpu time 0.01 seconds
system cpu time 0.00 seconds
memory 634.18k
OS Memory 16368.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 90 Switch Count 0

NOTE: There were 10000 observations read from the data set WORK.SENSITIVITY4.
NOTE: The data set WORK.SENSITIVITY4 has 10000 observations and 7 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 0.00 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds
memory 4995.68k
OS Memory 20744.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 91 Switch Count 0

NOTE: There were 10000 observations read from the data set WORK.SPECIFICITY4.
NOTE: The data set WORK.SPECIFICITY4 has 10000 observations and 7 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 0.01 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds
memory 4995.68k
OS Memory 20744.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 92 Switch Count 0

NOTE: There were 10000 observations read from the data set WORK.SENSITIVITY4.
NOTE: There were 10000 observations read from the data set WORK.SPECIFICITY4.
NOTE: The data set WORK.SENS_SPEC has 10000 observations and 10 variables.
NOTE: DATA statement used (Total process time):
real time 0.01 seconds
user cpu time 0.00 seconds
system cpu time 0.01 seconds
memory 1027.00k
OS Memory 16368.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 93 Switch Count 0

NOTE: There were 10000 observations read from the data set WORK.SENS_SPEC.
NOTE: The data set WORK.TEST has 4 observations and 4 variables.
NOTE: PROCEDURE FREQ used (Total process time):
real time 0.03 seconds
user cpu time 0.00 seconds
system cpu time 0.01 seconds
memory 1006.87k
OS Memory 16368.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 94 Switch Count 0

NOTE: There were 4 observations read from the data set WORK.TEST.
NOTE: PROCEDURE PRINT used (Total process time):
real time 0.01 seconds
user cpu time 0.00 seconds
system cpu time 0.01 seconds
memory 383.00k
OS Memory 16368.00k
Timestamp 17/03/2026 02:20:55 PM
Step Count 95 Switch Count 0

[n6b]

Wait a minute, I'm sorry if I've been confusing so let me state it clearly and correctly here.

We want to show that was have Sensitivity >= 77.5%

And we want to show that we have Specificity >= 77.5%

And we want to have 80% power to show that.

Thus the code in the macro has 77.5 and 22.5, but we want the output that you and I generated to be 80.00.

So the run showed 79.08% power with N=470.  So what I did was then increase the N to get the power up to 80%, which I achieved at N=480.

Then the next one I start on is Sensitivity = 85% and Specificity = 86%, and pick a new N to start with and then interate to get to 80% power.  But it'll take a long time that way.

[Rick_SAS]

What client or editor are you using to run this program? SAS EG? SAS Studio? The old Windows Display Manager?

From your description, it sounds like some resource is accumulating, which is why the process slows down. If you are seeing output to the log, an ODS output destination, or even the Results window in the old Display Manager. See Turn off ODS when running simulations in SAS - The DO Loop for a discussion and some tricks.

[n6b]

I'm using the old Display Manager.  I do have some ODS code, as you can see from where I posted the code for the entire macro, but I'm using that to output the results that I'm subsequently using.  Only a two-way proc freq and the four observation dataset generated by it is going to the output window.