Test prioritization: ideas
July 2019
After talking all week about prioritizing pages, elements, types of change, and variations, let’s finally talk about ideas themselves.
This whole series was sparked by a question from a reader, Antonio Duplessis:
Do you ever prioritize your ideas before taking action? (I assume so.)
Do you use a specific weighted matrix or framework?
Definitely. Sometimes. When working with several stakeholders, or seeking to scale a testing program, some sort of framework is a must.
It protects you from wasting effort on bad or impractical ideas, and it gives everyone visibility into what tests are on the horizon. And most importantly, it gives you a way to encode and communicate culture.
So many frameworks
There’s a ton of three-letter options out there for you - PIE, ICE, PXL. They all allow you to produce a numerical answer to a set of questions like “How hard will this be to build? How amazing do we expect the results to be? How sure are we about that?” Any one of these frameworks will work great.
My personal favorite approach comes from Pauline Marol of hotwire.com. You can read an overview in this blog post, or get more detail from this slide deck.
Her method involves asking a dozen or so yes/no questions about each test idea. Questions like “Does it make changes above the fold? Does it support one of our strategic goals? Is it on one of our top 3 high-traffic pages?”
This approach is great, because the answers to the questions are unambiguous. There’s nobody saying “🤔 I’d say this is about a 3 for effort.”
Is it worth building a custom framework?
Absolutely, you special snowflake you ❄. Your prioritization framework is a place for you to document your standards, team culture, strengths, weaknesses, and existing knowledge about the site you’re optimizing.
Standards
Is making data driven decisions important for your org? You can award points for ideas that are supported by other experiment results, analytics data, user testing, or other research methods. You can even award a bonus point for tests with multiple sources of supporting evidence.
Team culture
Striving for maximum output? Penalize ideas that require more than a few hours of development and QA.
Is creativity and originality important to your team? Give a point to ideas that are wildly different from anything else you’ve tried.
Strengths and weaknesses
Got 3 developers but only one designer? “No design” tests get a point.
Is there a stakeholder who often holds up or derails test launches? Award a point for ideas that have already been approved by this person.
Existing knowledge about the site
Your framework will grow as your testing program grows. In time you’ll learn things like “copy changes tend to move the needle more than design changes.” Once you’ve figured this out, you can start scoring copy changes higher.
Do you use a custom prioritization framework? In what ways does it encode your team culture? I’d love to hear about it; please hit Reply and share.