Cooking the books
August 2019
Over the past two days, we optimized tf out of our B2B lead generation site.
A prominent “Click Here for $1,000 Cash” button has driven engagement through the roof, and pointing all CTAs to the Contact Us form page has successfully increased the Form Page Visit conversion rate. 🏆
But did we really improve the site?
As ridiculous as these examples are, the answer is “maybe.”
Intuitively, we’d expect the “$1,000 Cash” CTA to frustrate visitors, as there’s no $1,000 to be had. There’s not even a mention of it on the form page where the CTA takes them.
Speaking of which, we’d expect most of the additional form page visitors to be more bewildered than motivated. They only landed there because they couldn’t find a way to get more info on the product.
Still.
Either the updates improved the site’s conversion rate, or they didn’t. We know we’ve got more engagement and form page visits; the stats calculator assures us of this fact.
The problem is that same stats calculator gravely informs us that, since we only get about 100 conversions a month, we need 6 weeks of data just to detect an effect of 30% or greater.
If these changes have driven conversions down to 75 a month, that is terrible - and we won’t know it. If they raised conversions to 120 a month, that is HUGE - and we won’t know it.
So how can we decide whether to keep our “$1,000” CTAs and keep sending all clickers straight to the contact form?