The mighty checklist
September 2019
After two days of judgmental diatribes against anyone who would launch an A/B in a hurry … here’s how to launch an A/B test in a hurry:
If you’re squinting on mobile, it’s a screenshot of 7 checklists, like this:
Yesterday I launched four tests in under an hour. One had an issue with traffic allocation - it had been adjusted during the QA process. But the checklist caught it.
Another one had a layout issue on IE that didn’t show up during QA … because the testing platform’s QA feature is kind of terrible. We caught that one within seconds, paused the test, and are working on a fix.
Having a thoroughly documented process for every stage of the test lifecycle lets you reduce errors, act with confidence, and move faster.
But here’s the thing: creating and maintaining these processes requires you to slow down.
It takes a lot of time to produce that original pile of checklists. And every time you use one, you must take the time to update your template as you go. A checklist is a living, breathing entity that you need to nurture with loving care.
If you’re in a hurry, you’ll skip the step where you keep the template up to date. So over time your checklist will be less and less relevant. Eventually you’ll start ignoring it.
Then you’ll be in a hurry, ignoring the checklist and skipping steps, and you’ll break the internet. (Or at least launch a broken test.)
So, no magic bullet. And no satisfying answer to the hurry-sick manager who wants these tests live ASAP so we can hit our target number of launches 😉
But with care and preparation, it is possible to move fast without breaking 💩.