Hi--
I'm putting together a new advertising creative campaign test - the creatives I'm using have never been tested before and the company has never down and creative test campaign. Normally, I would I would use proc power and base one proportion off a historical response rate and one proportion off an anticipated lift. So, like this
proc power;
TwoSampleFreq
Test=Fisher
Alpha =0.01 0.05 0.1
Sides = 1
GroupProportions = ( p1. p2.)
Power =.9
Npergroup =.;
The issue that I have is that I have no historical benchmark to base the sample size calculation off of; so, no prior response rate based on creatives how the creative performed in the past. What I will know is what the total universe of people I can pull a sample from will be. How should the sample size(s) be calculated when the creative test campaign is new?
This is what my test cells look like:
Customers | Tactic | Sample Size (n=) |
A. Hold Out Group | No Tactic | ? |
B. Control | Email Only | ? |
C. Test 1 | Direct Mail | ? |
D. Test 2 | Email + Direct Mail | ? |
Total (n=) | ? |
Any feedback/advice is greatly appreciated. Thanks!
One can always appeal to a "worst case scenario" Assume that your reference is 0.5, and generate sample sizes based on what you consider an important deviation from that value. Then the matter becomes balancing cost of conducting the survey versus the expected return.
Steve Denham
Hi Steve
Thank you for feedback. This was very helpful.
I have one more question if you have time
I would like to compute sample sizes based on these combinations:
A+B vs C+D
D vs A
D vs B
C vs A
C vs B
Should I use a different method/proc to calculate the sizes? Any further suggestions/advice of what to use is greatly
appreciated.
When in "worst case" mode where you have no prior estimate of any of the groups, it isn't going to make any difference (well except for the A+B vs C+D). Any pairwise comparison is the same. For instance if one group were at 0.5, and you wished to detect a shift of 0.05 with alpha=0.05 and power=0.8 using a two-sided Fisher's Exact test, you will need 1605 per group.
I would just modify your code to:
proc power;
TwoSampleFreq
Test=Fisher
Alpha =0.01 0.05 0.1
Sides = 1
GroupProportions = ( 0.5. 0.55.)
Power =.9
Npergroup =.;
run;
I get n per group values of 2635, 1747 and 1350 for the three alpha values given.
Steve Denham
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.
Find more tutorials on the SAS Users YouTube channel.