turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Sample Size for A/B/C/D test with no prior benchma...

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

05-19-2014 08:42 PM

Hi--

I'm putting together a new advertising creative campaign test - the creatives I'm using have never been tested before and the company has never down and creative test campaign. Normally, I would I would use proc power and base one proportion off a historical response rate and one proportion off an anticipated lift. So, like this

proc power;

TwoSampleFreq

Test=Fisher

Alpha =0.01 0.05 0.1

Sides = 1

GroupProportions = ( p1. p2.)

Power =.9

Npergroup =.;

The issue that I have is that I have no historical benchmark to base the sample size calculation off of; so, no prior response rate based on creatives how the creative performed in the past. What I will know is what the total universe of people I can pull a sample from will be. How should the sample size(s) be calculated when the creative test campaign is new?

This is what my test cells look like:

Customers | Tactic | Sample Size (n=) |

A. Hold Out Group | No Tactic | ? |

B. Control | Email Only | ? |

C. Test 1 | Direct Mail | ? |

D. Test 2 | Email + Direct Mail | ? |

Total (n=) | ? |

Any feedback/advice is greatly appreciated. Thanks!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to RobertNYC

05-20-2014 10:47 AM

One can always appeal to a "worst case scenario" Assume that your reference is 0.5, and generate sample sizes based on what you consider an important deviation from that value. Then the matter becomes balancing cost of conducting the survey versus the expected return.

Steve Denham

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SteveDenham

05-20-2014 10:58 AM

Hi Steve

Thank you for feedback. This was very helpful.

I have one more question if you have time

I would like to compute sample sizes based on these combinations:

A+B vs C+D

D vs A

D vs B

C vs A

C vs B

Should I use a different method/proc to calculate the sizes? Any further suggestions/advice of what to use is greatly

appreciated.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to RobertNYC

05-20-2014 11:11 AM

When in "worst case" mode where you have no prior estimate of any of the groups, it isn't going to make any difference (well except for the A+B vs C+D). Any pairwise comparison is the same. For instance if one group were at 0.5, and you wished to detect a shift of 0.05 with alpha=0.05 and power=0.8 using a two-sided Fisher's Exact test, you will need 1605 per group.

I would just modify your code to:

proc power;

TwoSampleFreq

Test=Fisher

Alpha =0.01 0.05 0.1

Sides = 1

GroupProportions = ( 0.5. 0.55.)

Power =.9

Npergroup =.;

run;

I get n per group values of 2635, 1747 and 1350 for the three alpha values given.

Steve Denham