turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Website AB testing: Determine time the test can re...

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

11-18-2015 10:28 AM

I have a website A/B test coming up. The test is this.

--I'm testing sale messaging content slot placement on a Home Page

--I have 2 versions of the Home Page created to show different audiences

-- I want to be able to determine "by noon" that the results will confirm my lift hypothesis. The information used to confirm this I think would be revenue generated from those who viewed the HP in their visit

Again, On the day a test launches, I want to know if I can determine by noon if it’s successful and statistically significant. I also want to be able to predict ahead of time how long something needs to run to be significant. I'm looking for advice on how to approach this. We use SAS in our office. For Direct mail campains I use a power anysis to calculate sample size but in this situation I’m not sure how to determine sample size and the length of time it will take to reach significance

From what I’ve been researching I am pretty sure power analysis should work but there are steps in the process im not sure I understand.

The literature I’ve been reading says:

- you'll need to pick an effect size (such the smallest difference between the pages that would make switching worth the cost) –
**I’m not sure what this means** - pick a a desired power threshold (which I would assume would be 0.8) – I understand that
- The Type I error rate for your test (0.05) – I understand that

The literature then says the power analysis will estimate the sample size needed to meet these conditions. **From there, you should be able to use page-view metrics to convert this estimate into time.** I’m not sure how to interpret that last statement.

For a DM A/B test I use this code for test and control groups.

```
proc power;
TwoSampleFreq
Test=Fisher
Alpha = 0.05
Sides = U
GroupProportions = ( /*Historical Benchmark: Response Rate*/ /*lift*/)
Power =.8
Npergroup =.;run;
```

Any examples, assistance with this or sample code will be greatly appreciated. Thanks!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to cara_catus

11-18-2015 01:31 PM

**From there, you should be able to use page-view metrics to convert this estimate into time.** I’m not sure how to interpret that last statement.

Basically looking at something like page hits per hour. If you get 100 hits per hour and you need 500 views for your use that translates to 5 hours. So if you need it by noon you need to start 5 hours prior.

Of course your page hits are possibly influenced by time of day but that may give you enough to start.

The effect size is going to be related on how you measure preference or whatever the two versions are supposed to garner. It might be difference between some sort of like scale (paired tests if each person views both pages) or the overall average scores. Something like "switch if new page has average score 0.4 or more larger on 1 to 10 scale than current". Not that I'm making any recommendation just an example from a planning or requirements document as an example.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to ballardw

11-18-2015 01:49 PM

This is great ballardw, thank you. I am still a little confused as how to determine the sample size or page hits. Using your example how would I determine that I need 500 views to reach statistical significance?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to cara_catus

11-18-2015 02:21 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to pearsoninst

11-18-2015 02:31 PM

hanks pearsoninst ! Real quick: out of the Proc Power examples on that website, which example would you suggest I follow for this

exercise.