👈 Home

ITP 2.2 and A/B testing - are we screwed?


Kind of.

How ITP 2.1 ruins your life

For Safari 12.1 users, client-side cookies are capped to a 7 day expiration.

So when a visitor lands on your site and your A/B testing platform sets a cookie to record which variation they’re in … that cookie goes away in a week.

If they come back within a week, the cookie’s still there, and expiration can pushed out another 7 days. Yay!

But if they come back in 8 days (or longer) they’ll be treated as a new visitor, and possibly bucketed into a different variation. Bad UX and muddy data.

Questions to ask yourself:

A) What percentage of site traffic is on Safari 12.1?

B) What percentage of visitors come to the site multiple times, at intervals longer than 7 days?

C) What percentage of conversions come from visitors who meet conditions A and B?

How ITP 2.2 ruins your life slightly more

Some client-side cookies will be capped at 24 hours.

If we change “7 days” to “1 day” in condition B above, it’s probably a much bigger number. Scary stuff.

So, which cookies?

It’s got nothing to do with your site or A/B testing vendor; it depends on how visitors get to your site. Specifically, cookies are capped at 24 hours when:

There’s no official list of these cross-site tracking domains, but surely we’re talking about social sites and ad platforms at a minimum. And most of your traffic from these sites will have query parameters 😔

What to do about it

Ask another set of questions:

D) What percentage of visitors are coming from a site that might be classified as “having cross-domain tracking capabilities” and landing on a URL with query parameters?

E) What percentage of conversions from those visitors happen on a second (or subsequent) visit, more than 24 hours after their first visit?

Add up C and E, and decide how much you care about all this.

There are fixes out there, but no simple ones yet. And because cross-domain and subdomain tracking are factors in which fix will work for you, it’ll be a custom job. Dev work will be involved.

For some businesses, the upshot of this analysis will be “It’s not worth A/B testing right now.”

Wanna chat more about #cookiepocalypse? Hit Reply. Let’s problem solve, or just commiserate.

Further reading

© 2022 Brian David Hall