👈 Home

How long will your A/B test take?


LOL nobody knows. Let’s take a hypothetical experiment, run it through some test duration calculators, and decide if we should be disturbed at the results.

The test

Say you’re running a simple A/B test (control + 1 variation). Here’s some info on the page you’re testing:

You’d like to be able to detect at least a 15% effect - so, raising the conversion rate from 6.5% to 7.5% or higher. (Or reducing it from 6.5% to 5.5% or lower, but let’s not talk about that.)

So that means we’ll choose:

We’ll go with the default settings common to all of these calculators:

How long will it take to run this test?

The results

Here are the estimated test durations from several calculators:

So … anywhere from 11 to 40 days.

What it all means

Different testing tools, and testing teams, use different methods to calculate the significance of test results, and therefore different methods to estimate test duration.

As long as you’re consistent - using the same method for both calculations - you’re probably safe from nasty surprises.

If you’re new to testing, or feel uneasy about statistics, just pick a tool and stick with it. Don’t let this stuff slow you down.

If you’re interested in going deeper into the details of these statistical methods, hit Reply and let me know. (No plans to start spamming this list with a bunch of equations; you have to opt in :)

© 2022 Brian David Hall