Test prioritization: changes
July 2019
Once you’ve decided what pages, and possibly what elements you’re optimizing, it’s time to think about how your variations will actually … vary.
There are four broad categories of changes you can make on your site. Here they are, in approximate order of highest-to-lowest impact. (Your mileage will vary.)
Presence / absence / position
Adding, removing, and rearranging elements is often the easiest and highest-impact change you can make on a page. What section performs best beneath the homepage hero? Do we even need this giant table of features? 🤔 What if the form was at the top of the page?
Design
Depending on your site and visitors, you’ll find design changes to have 🚀 high impact … or disappointingly low impact. Be bold in early tests so you can find out.
(Of course presence / absence / position changes are part of design, but here we’re talking about features like color, fonts, spacing, icons, and images. Go crazy with ‘em.)
Copy
Same warning as design! Either it really matters, or hardly anybody’s reading your copy. So make big changes to find out, then fine tune if you get encouraging results.
Functionality
Considering the effort that goes into implementing new functionality in an A/B test, it’s unlikely to be a high return type of change for your experimentation program.
Nothing like building out a quick shop feature, a shipping calculator, a multi-step form … and finding out that your site’s better off without it 😔
But it still beats building it out and leaving it on your site to damage conversions forever. So you should still test functionality when it’s based on compelling evidence, or it’s gonna get built anyway, or occasionally just for fun.
If you’ve got a dev on your team, there’s a good chance she’ll get bored if you only hand her show/hide and copy change tests. Let her build something cool, test it out, see what you can learn.