Power Simulation for a Two Proportions Test

BlueNose — Fri, 02 Jun 2017 21:18:36 GMT

Hello all,

I am trying to make a power analysis for a future clinical trial in which two proportions are to be compared. The analysis to be done is a non-inferiority analysis, i.e., it will be a one sided test for proportions difference, and the difference of proportions will be compared to a non-inferiority margin. The null hypothesis is P1-P2=-0.02, while the alternative hypothesis will be P1-P2>-0.02. The proportion in the first group (control) is assumed to be 8% while in the treatment group 6% (a smaller proportion is better).

I wrote this simulation, very much based on Rick Wicklin's example which I read in his book (that's why I publish this on this board, hoping that Rick will see this). The simulation ran and I got results, however, three problems occurred:

1. The results differ significantly from the results obtained from a sample size software, which used a formula rather than a simulation. The sample size in my simulation turned out to be much higher (about 300 subjects more overall).

2. I saved not only the P-Values of the non-inferiority test, but also the lower limit of the corresponding CI, which is a common practice which I saw in several articles. I expected the two methods to give the exact same results. But they don't. Not exactly.

3. I tried using PROC POWER to comapre the simulation results, but I keep getting an error saying: "NTOTAL is not available as a result option for TEST=." despite copying my code from the SAS documentation examples and changing the numbers only.

Can you kindly assist me in figuring out what is wrong with my simulation ?

Thank you in advance

%macro ODSOff(); 
	ods graphics off;
	ods exclude all;
	ods noresults;
%mend;

%macro ODSOn(); 
	ods graphics on;
	ods exclude none;
	ods results;
%mend;

/* Macro Variables */
%let NumSamples = 10000;  
%let nimargin = 0.02; /* Non-inferiority margin */
%let negnimargin = -0.02; /* Negative non-inferiority margin */
%let P_SOC = 0.08;    
%let P_Treatment = 0.06;  

/* 1. Simulating N samples of EACH group (Total = 2N) for each sample */
data PowerSizeSim(drop = i);
	call streaminit(321);
	Pc = &P_SOC;                                                        
	Pt = &P_Treatment;                                                      
	do N =  200 to 500 by 25 ;    /* N - sample size per group */
   		do SampleID = 1 to &NumSamples;
      		do i = 1 to N;
         		c = 1; x1 = rand("BERNOULLI", Pc); output;
         		c = 2; x1 = rand("BERNOULLI", Pt); output;
      		end;
   		end;
	end;
run;

data SimulatedData;
	set PowerSizeSim;
	if x1 = 0 then x2 = 1; else if x1=1 then x2 = 0;
	drop Pc Pt;
run;

/* 2. Compute Statistics */
%ODSOff 
proc freq data = SimulatedData;
	by N SampleID;
	table c*x2 / riskdiff(noninf margin=&nimargin method=fm) alpha=0.025;
	ods output PdiffNoninf = NITESTS;
run;
%ODSOn  


/* 3. Construct indicator var for obs that reject H0 */ 
data ResultsSize; 
	set NITESTS; 
	RejectH0 = (PValue <= 0.05);
	LowerCI = (LowerCL >= &negnimargin);
run; 

proc freq data=ResultsSize noprint; 
   by N; 
   tables RejectH0 / out = SimPower(where = (RejectH0 = 1));
   tables LowerCI /  out = SimPowerCI(where = (LowerCI = 1));
run;


/* 4. Output */
title2 'Simulated Power by Sample Size';
proc report data = SimPower;     
   column N PERCENT;
   define N / display 'Sample Size Per Group';            
   define PERCENT / display 'Simulated Power';
run;
title2;


/* PROC Power Validation */
proc power;
    twosamplefreq test = FM groupproportions = (0.08 0.06) nullproportiondiff = -0.02 alpha = 0.025
                  sides = U  power = 0.8 /*ntotal = .*/ NPERGROUP=.;
run;

Re: Power Simulation for a Two Proportions Test

Rick_SAS — Sat, 03 Jun 2017 00:49:07 GMT

The simulation part looks okay, so I would review the PROC FREQ / RISKDIFF(...) doc to make sure it is doing what you want. Also, why are you using alpha=0.025 in some places and PVALUE <= 0.05 in another?

Re: Power Simulation for a Two Proportions Test

BlueNose — Sat, 03 Jun 2017 09:28:38 GMT

Rick, thank you. I think you solved the mystery, well, one of them anyway. I specified alpha=0.025 in order to get a 95% CI, but I forgot to test the P-Value vs 0.025. I changed it now and the results of both the CI and the hypothesis testing are identical. The only thing that is still a mystery is why proc power doesn't want to work, but I guess that with a good simulation in hand, I don't really need it.

Re: Power Simulation for a Two Proportions Test

Rick_SAS — Mon, 05 Jun 2017 15:38:23 GMT

Unfortunately, I do not know PROC POWER well enough to advise you. But you could try posting just that question to the SAS Statistical Procedures Community.

topic Re: Power Simulation for a Two Proportions Test in SAS/IML Software and Matrix Computations

Power Simulation for a Two Proportions Test

Re: Power Simulation for a Two Proportions Test

Re: Power Simulation for a Two Proportions Test

Re: Power Simulation for a Two Proportions Test