A painted door (or “fake door”) test is a special type of de-risk it test that can save you tons of time and effort. Let’s look at what it is, when and how to use it, and what to watch out for.
What it is
A painted door test lets you gauge interest in a new feature, product, page, or piece of functionality … before you’ve built it. It involves updating your site with a link or callout that promises the new feature, then measuring clicks to see how many visitors would try it if it existed.
When to use it
You should use a painted door when the feature you’re considering will be expensive to implement, and you’re unsure about demand. (Don’t bother if you’re already so in love with the idea that you’ll build it no matter what.)
Suppose you sell artisanal beef jerky online, and you’re thinking about creating a subscription service. That’s a lot of work to set up; are you sure anybody wants it?
You can add a prominent “Subscribe and save!” button to your product pages, and measure the conversion rate on that button. At the end of a month, you’ll have a sense of how many visitors might sign up, and you can decide whether to proceed.
There are a couple ways this can go wrong.
What happens when I click the “Subscribe and save!” button – besides sending data to your analytics platform? If the answer is “nothing” then I’ll be irritated, think your site is broken, and possibly lose faith in your jerky.
Avoid this by setting up an email opt-in just for this project. When I click, let me know you’re working on this feature and offer to notify me as soon as it’s ready. (And follow up with these subscribers, no matter what you decide!)
A painted door test is less work than building the entire feature, but it’s still work. If you run the test and find yourself no closer to a decision, that’s a tragic failure.
It’s not always obvious how to decide, though. If you got 200 clicks in a month, should you build it? What if that’s 200 clicks out of 200,000 visitors (a measly 0.1% conversion rate)?
The answer will depend on the feature you’re considering, your team’s capacity and priorities, even existing conversion rates on the site. The single thing you must do is decide in advance what success looks like. Something like this:
We need at least 100 new subscribers a month for this to be worth the effort. Our current cart-to-checkout conversion rate is about 35%. If we assume subscription customers complete their signup at a comparable rate, we’ll need about 300 clicks on the button to get there.
With that in place, you can avoid analysis paralysis and make a confident decision. You’ll either be building a feature you know your visitors want, or you’ll save yourself the time and effort. Win win!