Website AB testing: Determine time the test can reach statistical significance using Proc Power

Occasional Contributor
Posts: 7

Website AB testing: Determine time the test can reach statistical significance using Proc Power

I have a website A/B test coming up. The test is this.

--I  have 2 versions of the Home Page created to show different audiences

-- I want to be able to determine "by noon" that the results will confirm my lift hypothesis. The information used to confirm this I think    would be revenue generated from those who viewed the HP in their visit

Again, On the day a test launches, I want to know if I can determine by noon if it’s successful and statistically significant. I also want to be able to predict ahead of time how long something needs to run to be significant. I'm looking for advice on how to approach this. We use SAS in our office. For Direct mail campains I use a power anysis to calculate sample size but in this situation I’m not sure how to determine sample size and the length of time it will take to reach significance

From what I’ve been researching I am pretty sure power analysis should work but there are steps in the process im not sure I understand.

The literature I’ve been reading says:

1. you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost) – I’m not sure what this means
2. pick a a desired power threshold (which I would assume would be 0.8) – I understand that
3. The Type I error rate for your test (0.05) – I understand that

The literature then says the power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.

For a DM A/B test I use this code for test and control groups.

``````proc power;
TwoSampleFreq
Test=Fisher
Alpha = 0.05
Sides = U
GroupProportions = ( /*Historical Benchmark: Response Rate*/ /*lift*/)
Power =.8
Npergroup  =.;run;
``````

Any examples, assistance with this or sample code will be greatly appreciated. Thanks!

Super User
Posts: 11,773

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.

Basically looking at something like page hits per hour. If you get 100 hits per hour and you need 500 views for your use that translates to 5 hours. So if you need it by noon you need to start 5 hours prior.

Of course your page hits are possibly influenced by time of day but that may give you enough to start.

The effect size is going to be related on how you measure preference or whatever the two versions are supposed to garner. It might be difference between some sort of like scale (paired tests if each person views both pages) or the overall average scores. Something like "switch if new page has average score 0.4 or more larger on 1 to 10 scale than current". Not that I'm making any recommendation just an example from a planning or requirements document as an example.

Occasional Contributor
Posts: 7

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

This is great ballardw, thank you.  I am still a little confused as how to determine the sample size or page hits. Using your example how would I determine that I need 500 views to reach statistical significance?

Frequent Contributor
Posts: 108

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

Hi ,
http://www.ats.ucla.edu/stat/sas/dae/t_test_power2.htm
Occasional Contributor
Posts: 7