What does an A/B test look like, anyway?

You know A/B testing is the right thing to do, right? Do you know what a good A/B test looks like? No? Well, thinks is here to help.

GrokDotCom shares Nielsen data every month, showing the top converting websites. Amazingly, online flower and gift retailer ProFlowers converts a whopping 43% of its visitors to sales. While the nature of its customers – such as husbands who need a last-minute gift when they forgot their anniversary – certainly influences that number, 43% is huge by any standard. Not surprisingly, ProFlowers uses A/B testing regularly. For instance, during the recent Mother’s Day holiday, ProFlowers ran these four separate landing pages against one another:

(click for a much larger version: 1607 x 457 pixels; about 1MB)

If you look closely, you can see they only changed the middle section – below the header and above the product listings – trying different versions to drive improved conversion. This image calls out what changed:

(click for a much larger version: 1607 x 457 pixels; about 1MB)

So, what did they test?

• Messages – Two of the four pages (the champion and the first challenger) focus more on savings. Note how prominent the “Free Glass Vase” and “Featured Offer” are in the first two. By contrast, the second pair focus more on ProFlowers’ quality, making the Guarantee message more prominent in each.
• Layout – Take note of where the “Free Glass Vase” and Guarantee messages appear on the landing pages.

This is very consistent with our list of what to test first and the 6 simple rules of online/offline merchandising, but shows it in a real world context.

Since ProFlowers ran this as a limited-time test, it’s likely you’ll see the winning design more prominent in their future landing pages. At the same time, don’t be surprised if you don’t. ProFlowers didn’t get to a 43% conversion rate by resting on their… laurels. Their most recent promotion is also a test, showing that ProFlowers using a continuous improvement process to further increase conversion. So, here’s the real question: why aren’t you?

Are you getting enough value out of your small business website? Want to make sure your business makes the most of the local, mobile, social web? thinks helps you understand how to grow your business via the web, every day. Get more than just news. Get understanding. Add thinks to your feed reader today.

Or subscribe via email.

And while you’re at it, don’t forget to follow Tim on Twitter.

Technorati Tags:
a/b testing, bounce rate, call-to-action, e-marketing, internet business, internet marketing, landing page design, landing page optimization, marketing, measurement, metrics, online marketing, page design, performance, solutions, strategy, testing, tools