An A/B test can't tell you why
May 2019
Yesterday we looked at why hypotheses might be unnecessary, or at the very least insufficient, to run a successful testing program. But there’s one type of hypothesis that will actively damage your program - the hypothesis with “why.”
Here’s a useless hypothesis:
If we change the CTA from Orange to Blue, it will increase conversions.
… and here is its pernicious sibling:
If we change the CTA from Orange to Blue, it will increase conversions, because blue is a soothing color.
You can’t know this! Sorry, not sorry. Let’s look at why you can’t know, and why not knowing doesn’t matter.
We’re really good at coming up with reasons
Take a look at these 3 hero images from an email signup page used in Obama’s 2018 presidential campaign:
(You can read more about the Obama experiment on Optimizely’s blog.)
Which one got more email signups? And why?
I’ve shown these images to dozens of smart people and heard dozens of great explanations for why each one won.
Here’s the actual winner:
Did it win because Americans in 2008 were concerned with family values? Because Obama looks friendlier? Because the red button stands out more on the light background? We will never know.
We rarely even know our own reasons
The fact is, we make choices for a combination of reasons. If you use your imagination, you can probably produce at least 3 different explanations for what you ordered the last time you ate in a restaurant.
To further complicate matters, we don’t fully understand all the reasons we do things. (So yeah, there are 5 other subconscious factors that led you to order the Monte Cristo.) There’s plenty of research to support this assertion, but you’re busy - so here’s a New York Times article with lots of nice links.
… but it doesn’t matter
It’s a bummer that we can’t know whether this image won because of the color scheme or the family. But knowing that we can’t know … sets us free.
Instead of making a false assumption about what this test taught us and asking “Where else can we test pictures of his family?” we can focus on what the test did teach us:
Changing this background image has a huge impact on the conversion rate.
This is good evidence that we should continue to test the image on this page. It’s decent evidence that we should test images in other places that have the traffic and conversions to run experiments. This is a powerful insight! There are plenty of sites where changing images has no effect whatsoever.
We don’t have to stick with whitewashed images, or family images. We don’t have to rewrite all our copy to mention his wife and kids. We’ve run a test, improved the conversion rate, and measured the impact of a particular type of change on a particular element on a particular page. That’s how it starts. No need to ask why.