More scary stories
October 2019
Yesterday I mentioned the frightening possibility that your A/B test variations represent an underwhelming, arbitrary subset of all possibilities.
Oneâs better than the rest, but theyâre all worse than whatâs truly best.
And if youâre using always-on personalization, your site will dutifully choose one subpar variation or another for each visitor. Forever.
Hereâs more nightmare fodder: what if the type of change youâre testing ⌠barely matters?
For example, on your product page, youâve decided to test highlighting different benefits in the hero section. The idea being, each visitor will connect best with one key benefit; if you can put that benefit front and center then theyâll be more likely to convert.
Yesterdayâs cautionary tale would warn you that you might be missing a key benefit, or phrasing a benefit wrong, so your site never reaches its full potential.
Forget that; what if benefits are the wrong thing to put on this page, in the hero, period?
Suppose most visitors to the product page are more concerned with price details, or whether they trust your company. There is no âoptimal benefitâ to display for each visitor. At least not here.
So while your personalization engine labors to choose one, and your stats dashboard tells you how many visitors saw Benefit A vs Benefit B, and you try to parse out whether that means youâre winning or not ⌠the whole thing is a sham.
You picked the wrong variable to tweak, and youâll never know. Instead youâll show up at meetings and report âIt seems that Benefit D resonates most strongly with mobile visitors on the east coast who came from a Google search.â
Sweet dreams!