Does visitor count matter?
June 2019
Not all that much. Not nearly as much as conversion count. Here, let’s prove it.
What’s the impact of visitor count?
Suppose you’ve got 1,000,000 monthly visitors to your store, but only 500 conversions. So, a 0.05% conversion rate. Not great, but with all those visitors, surely you can run a ton of tests, right?
So let’s assume we’re running a 5 variation test for 4 weeks. How much of a lift will you be able to detect?
About 70% - good to know; we’ll have to test pretty dramatic changes. (Screenshots from the AB+ Test Calculator by ConversionXL “Pre-Test Analysis” tool.)
Now suppose you don’t have the luxury of so many visitors - say you’re getting 100,000 instead of 1 million. This probably tanks your testing program, right?
… Not so much. Cutting the traffic down by 1/10th barely impacts the effect we can measure in a month; it’s still about 70%.
Cut it down again, from 100,000 to 10,000 monthly visitors, and you’re still looking at ~70% detectible effect:
In fact, it’s not till your monthly visitor count drops to 1,000 that we see a noticeable impact on our test’s ability to detect lift:
… and somehow, fewer visitors improved your test’s ability to detect changes.
What’s going on here?
Conversions are the signal that your test measures. Non-converting visitors are just noise. So adding more of them makes your data set noisier, and your tests less effective.
On the other hand, if you can seriously reduce the size of the audience you’re addressing, without sacrificing the number of converting visitors, your test is more sensitive. You’ll get cleaner results, faster.
What you can do today
Become low key obsessed with this mythical audience of “converting visitors.” What can you learn about their behavior? What pages do they visit, what elements do they scroll to or interact with? Focus your efforts there.
This will also point to insights about non-converting visitors. Are they employees, existing customers, bots? Nothing to worry about in small numbers, but if a sizable majority of visitors fall in the “never gonna convert” bucket, it’s worth excluding them from experiments.
More than anything, keep in mind that it’s conversions, not visitors, that will determine the testing capacity of your site. Happy testing!