Dealing with pressure to call tests early (1/3)
May 2019
You already know that calling a test too early is a Bad Thing. It can lead to Type I and Type II errors, leaving your conversion rates worse than they could’ve been, or even worse than where they started. But not everyone around you sees it the same way, and sometimes well-meaning people on your team will lean on you to call a test and move on. So how do you change their minds?
The first step is to understand where they’re coming from. Do they just need a stats lesson? Or do they have information you lack? Maybe you should call the test this time 🤔
Here’s a quick list of some of the likely offenders, their possible motivations, and what you need to learn about them in order to have a productive conversation about test stopping criteria (and greater matters).
Management / Executives
It’s likely that you have more statistical training than the exec asking why you don’t just call that test now, but they care about numbers just as much as you. At a minimum, they’re trying to hit quarterly targets; maybe monthly, possibly even weekly.
Do you know what those targets are? Is the team on track, or close, or way off course? Find a way to have a conversation about numbers with these folks. “How are you measuring success?” is usually all you need to say. Then listen actively, ask clarifying questions, take some notes. Show you care. When it comes time for them to listen to you talk about experimentation, you’ll have a more agreeable audience.
Clients
The conversation with a client is similar to with an executive (“How are you measuring success?”) but with the additional caveat that you have way less perspective on all that’s going on with them. There may be an internal initiative that conflicts with a currently-running test; they may be receiving downward pressure to “test aggressively” - meaning someone is insisting that they launch 10 tests this quarter. (A terrible metric, but let’s save it for another day.)
Media Buyers
If you’re testing on paid search landing pages, the situation is even more complicated and the pressure is more intense. Media buyers are pouring money into a hole and watching it disappear - if they think you’ve got a winning landing page, of course they want to drive all their traffic there, starting yesterday.
So you can empathize with them, but you still don’t want to call a test after 5 days and 43% statistical significance. What can you do?
For now, find an opportunity to learn more about how they make decisions with data. How many campaigns are they running for a given audience? How do they test out new ad copy? What’s been their biggest win in the last few months? What was an ad they thought would do well that just tanked? What were the numbers there?
You likely have a lot to learn from each other, and it may be up to you to initiate that exchange. See what you can find out about their process, their day-to-day view on data, and how they make decisions. For now, just listen.
What you can do today
Figure out what meeting you need to drop in on, what 1-to-1 you need to schedule, what questions you need to ask, or where you need to sit at lunch in order to learn more about your team and collaborators’ point of view on making decisions with data. Can you do a working session with a media buyer? Does the VP of Marketing have 10 minutes to walk you through the metrics behind this quarter’s OKRs? Can you lead off your next client calls asking about their internal success metrics? People generally love to be asked about this stuff; don’t be shy.
Tomorrow we’ll look at some tools and evidence you can use to support the case for running tests to completion, and the following day we’ll put it together into a story and talking points. Happy testing!