Differentness
July 2019
“Differentness” refers to how much two test variations differ from each other, or how much a group of test variations collectively differ from one another. It’s the thing you should maximize when designing tests.
Having thrown around this idea in discussing where ideas come from and how to prioritize variations, I thought it deserved its own entry.
You can’t measure it
Unless you’re doing something silly like optimizing border radius, the differentness between your variations will be subjective.
That’s okay, because it can still be perceived and discussed. Argued about, even. And the outcome of that argument could lead to a variation that’s even differenter - which is great. Because the saddest outcome in A/B testing is looking at inconclusive test results and saying “we need to test bigger changes 😢.”
It depends on the type of changes you’re making
Variations in a full page redesign will look different in obvious ways - length, use of color, inclusion/exclusion of entire sections, images and icons … they’ll be all over the place.
But when you’re testing copy, differentness is subtler. A variation that adds a subheading where there wasn’t one before … is actually quite different. So are two headlines that present distinct benefits of a product. Or one with a formal, factual tone versus a playful, informal one. (Whether your visitors pick up on this difference is something you have to test.)
Have the talk
So differentness can save you from wasted test cycles and despair, yet it can’t be objectively measured. What should you do, then?
Just be sure your team is talking about it. “Is this variation different enough? Could we take it even further? What would a completely opposite approach look like?”
Having these discussions will put you light years ahead of teams who only ever discuss what they think will win. (Or, later, why they think things didn’t win.) It’ll help you test smarter, and not fall in love with your ideas.
Tomorrow, we’ll look at one of the dreadful consequences of falling short in the differentness department. Until then, happy optimizing 👋.