# How long will your A/B test take?

June 2019

LOL nobody knows. Let’s take a hypothetical experiment, run it through some test duration calculators, and decide if we should be disturbed at the results.

### The test

Say you’re running a simple A/B test (control + 1 variation). Here’s some info on the page you’re testing:

- 1,000 visitors a day
- 6.5% conversion rate

You’d like to be able to detect at least a 15% effect - so, raising the conversion rate from 6.5% to 7.5% or higher. (Or reducing it from 6.5% to 5.5% or lower, but let’s not talk about that.)

So that means we’ll choose:

- 15% minimum detectable effect

We’ll go with the default settings common to all of these calculators:

- 95% confidence
- 80% statistical power

How long will it take to run this test?

### The results

Here are the estimated test durations from several calculators:

- AB Testguide: 11 days
- VWO: 20 days
- ConversionXL: ~40 days
- Optimizely: 19 days
- Evan Miller: 20 days
- Adobe: 22 days

So … anywhere from 11 to 40 days.

### What it all means

Different testing tools, and testing teams, use different methods to calculate the significance of test results, and therefore different methods to estimate test duration.

As long as you’re *consistent* - using the same method for both calculations - you’re probably safe from nasty surprises.

If you’re new to testing, or feel uneasy about statistics, just pick a tool and stick with it. Don’t let this stuff slow you down.

If you’re interested in going deeper into the details of these statistical methods, hit Reply and let me know. (No plans to start spamming this list with a bunch of equations; you have to opt in :)