Programming the statistical procedures from SAS

Sample Size for A/B/C/D test with no prior benchmarks set

Reply
Frequent Contributor
Posts: 98

Sample Size for A/B/C/D test with no prior benchmarks set

Hi--

I'm putting together a new advertising creative campaign test  - the creatives I'm using have never been tested before and the company has never down and creative test campaign.  Normally, I would I would use proc power and base one proportion off a historical response rate and one proportion off an anticipated lift. So, like this

proc power;

  TwoSampleFreq

  Test=Fisher

  Alpha =0.01 0.05 0.1

  Sides = 1  

  GroupProportions = ( p1. p2.)

  Power =.9

  Npergroup  =.;

The issue that I have is that I have no historical benchmark  to base the sample size calculation off of; so,  no prior response rate based on creatives how the creative performed in the past. What I will know is what the total universe of people I can pull a sample from  will be. How should the sample size(s) be calculated when the creative test campaign is new? 

This is what my test cells look like:

CustomersTactic Sample Size (n=)
A. Hold Out GroupNo Tactic?
B. ControlEmail Only?
C. Test 1Direct Mail ?
D. Test 2Email + Direct Mail?
Total (n=)?

Any feedback/advice is greatly appreciated. Thanks! 

Respected Advisor
Posts: 2,655

Re: Sample Size for A/B/C/D test with no prior benchmarks set

One can always appeal to a "worst case scenario"  Assume that your reference is 0.5, and generate sample sizes based on what you consider an important deviation from that value.  Then the matter becomes balancing cost of conducting the survey versus the expected return.

Steve Denham

Frequent Contributor
Posts: 98

Re: Sample Size for A/B/C/D test with no prior benchmarks set

Hi Steve

Thank you for feedback. This was very helpful.


I have one more question if you have time Smiley Happy

I would like to compute sample sizes based on these combinations:

A+B vs C+D

D vs A

D vs B

C vs A

C vs B


Should I use a different method/proc to calculate the sizes?   Any further suggestions/advice  of what to use is greatly

appreciated.

Respected Advisor
Posts: 2,655

Re: Sample Size for A/B/C/D test with no prior benchmarks set

When in "worst case" mode where you have no prior estimate of any of the groups, it isn't going to make any difference (well except for the A+B vs C+D).  Any pairwise comparison is the same.  For instance if one group were at 0.5, and you wished to detect a shift of 0.05 with alpha=0.05 and power=0.8 using a two-sided Fisher's Exact test, you will need 1605 per group.

I would just modify your code to:

proc power;

  TwoSampleFreq

  Test=Fisher

  Alpha =0.01 0.05 0.1

  Sides = 1  

  GroupProportions = ( 0.5. 0.55.)

  Power =.9

  Npergroup  =.;

run;

I get n per group values of 2635, 1747 and 1350 for the three alpha values given.

Steve Denham

Ask a Question
Discussion stats
  • 3 replies
  • 345 views
  • 3 likes
  • 2 in conversation