The second-worst outcome in A/B testing
August 2019
… is “all variations performed worse than control; further testing necessary … maybe.”
This one hurts. You’ve invested the time and money into running an experiment, and your results prove that the change you were testing does in fact impact conversions.
But you don’t have a winner, and you don’t have a clear path forward.
The dilemma
Is your control truly the optimal experience? If so, you should stop testing here.
Or did you just test the wrong changes? If so, you should try again with a different approach.
How to decide
How many variations did you test, and how different were they?
If it’s just one variation that lost, you should keep trying - but try more variations, and make them as different as possible.
If that’s exactly what you did, and they all lost, move on.
Sure, maybe there’s some other mysterious variation that would beat your control. Maybe you’ll even find it … one day. But for now, you have an experience that beats 5-10 others; you’re already in the place you hoped to reach through experimentation.
Working backwards
You can avoid the heartrending dilemma that is the second-worst outcome in A/B testing, and the bewildering quandary that is the worst outcome in A/B testing, just by testing more variations.
Enough to give you confidence that if they all lose, or just fail to win, it means you’re fine sticking with control. Because “we found that the control experience is optimal” 🏆 is actually a great outcome.