topic Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power in Statistical Procedures

Website AB testing: Determine time the test can reach statistical significance using Proc Power

cara_catus — Wed, 18 Nov 2015 15:28:29 GMT

I have a website A/B test coming up. The test is this.

--I'm testing sale messaging content slot placement on a Home Page

--I have 2 versions of the Home Page created to show different audiences

-- I want to be able to determine "by noon" that the results will confirm my lift hypothesis. The information used to confirm this I think would be revenue generated from those who viewed the HP in their visit

Again, On the day a test launches, I want to know if I can determine by noon if it’s successful and statistically significant. I also want to be able to predict ahead of time how long something needs to run to be significant. I'm looking for advice on how to approach this. We use SAS in our office. For Direct mail campains I use a power anysis to calculate sample size but in this situation I’m not sure how to determine sample size and the length of time it will take to reach significance

From what I’ve been researching I am pretty sure power analysis should work but there are steps in the process im not sure I understand.

The literature I’ve been reading says:

you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost) – I’m not sure what this means
pick a a desired power threshold (which I would assume would be 0.8) – I understand that
The Type I error rate for your test (0.05) – I understand that

The literature then says the power analysis will estimate the sample size needed to meet these conditions. From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.

For a DM A/B test I use this code for test and control groups.

proc power;
  TwoSampleFreq
  Test=Fisher
  Alpha = 0.05
  Sides = U  
  GroupProportions = ( /*Historical Benchmark: Response Rate*/ /*lift*/)
  Power =.8
 Npergroup  =.;run;

Any examples, assistance with this or sample code will be greatly appreciated. Thanks!

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

ballardw — Wed, 18 Nov 2015 18:31:32 GMT

From there, you should be able to use page-view metrics to convert this estimate into time. I’m not sure how to interpret that last statement.

Basically looking at something like page hits per hour. If you get 100 hits per hour and you need 500 views for your use that translates to 5 hours. So if you need it by noon you need to start 5 hours prior.

Of course your page hits are possibly influenced by time of day but that may give you enough to start.

The effect size is going to be related on how you measure preference or whatever the two versions are supposed to garner. It might be difference between some sort of like scale (paired tests if each person views both pages) or the overall average scores. Something like "switch if new page has average score 0.4 or more larger on 1 to 10 scale than current". Not that I'm making any recommendation just an example from a planning or requirements document as an example.

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

cara_catus — Wed, 18 Nov 2015 18:49:56 GMT

This is great ballardw, thank you. I am still a little confused as how to determine the sample size or page hits. Using your example how would I determine that I need 500 views to reach statistical significance?

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

pearsoninst — Wed, 18 Nov 2015 19:21:08 GMT

Hi ,
I think , the below link will help you ..
http://www.ats.ucla.edu/stat/sas/dae/t_test_power2.htm

Re: Website AB testing: Determine time the test can reach statistical significance using Proc Power

cara_catus — Wed, 18 Nov 2015 19:31:21 GMT

hanks pearsoninst ! Real quick: out of the Proc Power examples on that website, which example would you suggest I follow for this

exercise.