Is it okay to run winning variations at 100%?
February 2020
Of course! In fact, Google Optimize even has a button you can click to do exactly that:
If the data tells you that one of your test variants is making you more money than the rest, you should start serving it to all visitors as soon as you can.
But.
You should also open a ticket for your dev team to put the winning experience into production.
Leaving an A/B test variation running at 100% for too long can bring problems.
Too many tests
If youâre using the free version of Google Optimize, youâre limited to 5 live experiments at a time.
So if youâve got 3 â100% Experienceâ experiments running, youâre down to 2 live experiments.
This isnât the end of the world. Ideally youâre running fewer experiments with more variations, optimizing instead of just validating ideas.
But sooner or later youâll hit your limit.
Test complexity
Now that the official âControlâ experience on your site consists of âoriginal website + some 100% Experience code,â your A/B test engineer will have to code new tests against that.
And sheâll be fine, but it does add complexity. Here are the scenarios sheâll have to allow for during development and QA:
- Page loads, 100% Experience code runs, then new test code runs
- Page loads, new test code runs, then 100% Experience code runs
Not too terrible, but literally twice as complex as the case where thereâs no 100% Experience. And subject to random effects, since you canât control the timing.
And ⌠when you layer on multiple 100% Experiences, the bugs just multiply.
The SEO boogeyman will come to get you
Google loves A/B testing, of course. It says it right there in the Optimize and Google Search support article:
Pretty unambiguous. But scroll down a bit:
Uh-oh, what does that mean?
As with all things SEO, the precise answer is apparently unknowable. But if youâre going against Googleâs stated best practices, even if youâre using a Google product to do so, youâre at risk.
The risk of negative impact on your SEO rankings is uncertain. But the risk of feeling like an idiot if it happens ⌠thatâs pretty much guaranteed.
Program health
The last consideration is not a problem caused by 100% Experiences, but a problem they might indicate.
Why canât you just put the winning variation into production?
If youâre on a temporary code freeze, or your web developer is on vacation, no worries.
But if youâre struggling to get organizational buy-in, even with a winning test result, youâre in trouble.
If your website is such a mess and updates are so expensive that you canât afford them, thatâs a problem.
And as exciting and lucrative as experimentation is, you might need to hit the Pause button until youâve got the support you need to take winning tests across the finish line.