Rum raisin or bubblegum 🍨?
October 2019
Well, which one?
I worked at Baskin Robbins in high school, and these were the two flavors we couldn’t give away.
Luckily, there were 29 other flavors to choose from, so most people made out okay.
When you test different experiences on your site, you’re offering choices to your audience as a whole.
But … here’s the 💩 that keeps me up at night … what if you’ve arbitrarily chosen bad experiences to test?
What if your visitors were super excited to get some ice cream, but are kind of grossed out by the flavors you’ve picked?
You might see twice as many choosing bubblegum as rum raisin - a significant result! 📈 But you’ll never know how many scoops of butter pecan you could’ve sold, if only it’d been an option.
You can mitigate this risk in testing by testing a wide range of alternatives. (It’s easy to name 2 unpopular ice cream flavors; 6 would be a challenge.)
The scary bit is when you instrument your site for “always-on” personalization.
The 🤖 will work hard to choose one flavor for each visitor. Rum raisin for you, bubblegum for you. But your visitors will be forever underwhelmed, dreaming of rocky road.