Painted door disasters
July 2019
In a previous post, I wrote about using “painted door” tests to gauge interest in an expensive or risky new feature. That article called out some, but not all, of the risks that come with this kind of test.
The “shiny” effect
A reader, Tim Duke, commented:
You forgot to mention that sometimes people walk through painted doors just because the doors look neat (“Oh look, a new big red button, I’ll click it”).
Absolutely. This effect could lead to the disastrous scenario where you build a worthless feature thinking that you’ve already proved its value. 🤦♂️
Sorry, what door?
He goes on:
Conversely, if you hide your door in camouflage and nobody goes through it, is it the feature that nobody wants or is it that the door wasn’t visible?
Terrifying but totally true. In your efforts to counteract the “shiny” effect, what if you make the new feature link so subtle that no one can find it? 🤦♂️
What to do
One way to mitigate both of these risks is to use an element you’ve used before, and have some data on. Like a promo banner at the top of the site, or a “50% scroll” modal window.
Suppose that 8% of visitors clicked on the promo banner for a free shipping code last March, but only 5% clicked to learn more about your “Shoes for the needy” program in Q2 😔.
If you know that, you’ve got some guidelines for the popularity of your (proposed) new Virtual Shoe Sizer app. You know that 10% of visitors is a pretty high number, and 4% is pretty low.
Whatever results you get, the experiment is just one piece of evidence. Of course you’ll talk to some actual users and survey the competitive landscape before you start building the Virtual Shoe Sizer.
Be careful out there, and happy testing.