Yesterday I mentioned the frightening possibility that your A/B test variations represent an underwhelming, arbitrary subset of all possibilities.
One’s better than the rest, but they’re all worse than what’s truly best.
And if you’re using always-on personalization, your site will dutifully choose one subpar variation or another for each visitor. Forever.
Here’s more nightmare fodder: what if the type of change you’re testing … barely matters?
For example, on your product page, you’ve decided to test highlighting different benefits in the hero section. The idea being, each visitor will connect best with one key benefit; if you can put that benefit front and center then they’ll be more likely to convert.
Yesterday’s cautionary tale would warn you that you might be missing a key benefit, or phrasing a benefit wrong, so your site never reaches its full potential.
Forget that; what if benefits are the wrong thing to put on this page, in the hero, period?
Suppose most visitors to the product page are more concerned with price details, or whether they trust your company. There is no “optimal benefit” to display for each visitor. At least not here.
So while your personalization engine labors to choose one, and your stats dashboard tells you how many visitors saw Benefit A vs Benefit B, and you try to parse out whether that means you’re winning or not … the whole thing is a sham.
You picked the wrong variable to tweak, and you’ll never know. Instead you’ll show up at meetings and report “It seems that Benefit D resonates most strongly with mobile visitors on the east coast who came from a Google search.”