topic Sample Size for A/B/C/D test with no prior benchmarks set in Statistical Procedures

Sample Size for A/B/C/D test with no prior benchmarks set

RobertNYC — Tue, 20 May 2014 00:42:12 GMT

Hi--

I'm putting together a new advertising creative campaign test - the creatives I'm using have never been tested before and the company has never down and creative test campaign. Normally, I would I would use proc power and base one proportion off a historical response rate and one proportion off an anticipated lift. So, like this

proc power;

TwoSampleFreq

Test=Fisher

Alpha =0.01 0.05 0.1

Sides = 1

GroupProportions = ( p1. p2.)

Power =.9

Npergroup =.;

The issue that I have is that I have no historical benchmark to base the sample size calculation off of; so, no prior response rate based on creatives how the creative performed in the past. What I will know is what the total universe of people I can pull a sample from will be. How should the sample size(s) be calculated when the creative test campaign is new?

This is what my test cells look like:

Customers	Tactic	Sample Size (n=)
A. Hold Out Group	No Tactic	?
B. Control	Email Only	?
C. Test 1	Direct Mail	?
D. Test 2	Email + Direct Mail	?
Total (n=)		?

Any feedback/advice is greatly appreciated. Thanks!

Re: Sample Size for A/B/C/D test with no prior benchmarks set

SteveDenham — Tue, 20 May 2014 14:47:34 GMT

One can always appeal to a "worst case scenario" Assume that your reference is 0.5, and generate sample sizes based on what you consider an important deviation from that value. Then the matter becomes balancing cost of conducting the survey versus the expected return.

Steve Denham

Re: Sample Size for A/B/C/D test with no prior benchmarks set

RobertNYC — Tue, 20 May 2014 14:58:57 GMT

Hi Steve

Thank you for feedback. This was very helpful.

I have one more question if you have time

I would like to compute sample sizes based on these combinations:

A+B vs C+D

D vs A

D vs B

C vs A

C vs B

Should I use a different method/proc to calculate the sizes? Any further suggestions/advice of what to use is greatly

appreciated.

Re: Sample Size for A/B/C/D test with no prior benchmarks set

SteveDenham — Tue, 20 May 2014 15:11:31 GMT

When in "worst case" mode where you have no prior estimate of any of the groups, it isn't going to make any difference (well except for the A+B vs C+D). Any pairwise comparison is the same. For instance if one group were at 0.5, and you wished to detect a shift of 0.05 with alpha=0.05 and power=0.8 using a two-sided Fisher's Exact test, you will need 1605 per group.

I would just modify your code to:

proc power;

TwoSampleFreq

Test=Fisher

Alpha =0.01 0.05 0.1

Sides = 1

GroupProportions = ( 0.5. 0.55.)

Power =.9

Npergroup =.;

run;

I get n per group values of 2635, 1747 and 1350 for the three alpha values given.

Steve Denham