A/B tested landing page headlines for 2 weeks straight

My client was stuck with a 2% conversion rate on their service page, so I ran a split test between a benefit-driven headline and a question-based one. The question headline pulled in 4.2% conversions over 1,200 visits. Has anyone else seen big jumps just from changing one sentence?

3 comments

3 Comments

the_ruby1mo ago

That's actually a really good point about returning visitors and I've seen the same thing happen. The new visitor bias is real and it can mess with your results if you only look at two weeks of data. When you pull in a bunch of fresh traffic through ads or social media those people are already warmed up to take action, so the question headline grabs them fast. But if your service is something people need to think about or research before buying, you gotta look at what happens when they come back. That 1% drop on returning visitors could mean the question made them think too hard or feel pressured, and that killed the trust they had. Running tests for a month is the way to go because you catch those week over week patterns and see if the headline holds up for repeat shoppers too.

wood.faith1mo ago

My plumber test showed that exact same returning visitor drop too.

the_anthony1mo ago

Wait, did you track what happened with returning visitors versus new ones in that test? I had a similar situation a few years back where I was testing headlines for a local plumber's site. The question headline killed it for first time visitors, like 5% conversion compared to 2.5%. But here's the weird part - people who came back to the site actually converted way less on the question headline, like 1%. So overall it balanced out to almost the same as the boring headline. I always tell people to run tests for at least a month because the two week window can trick you with new visitor bias. What kind of traffic was driving those 1200 visits?