Ultimate A/B testing case study roundup (Small Business E-commerce Link Digest – June 25, 2010)

It’s no secret here that I believe A/B testing is one of the the best ways to help your customer and quiet Avinash Kaushik’s HIPPOs (see Rule #10) (Full Disclosure: As the head of e-commerce, I can be something of a HIPPO myself in my day job. So, in some ways, I know the dangers even more. It’s easy to think you’re really smart when you’ve been doing something for a while – and really cool when a test a.) proves you wrong and b.) makes you a lot of money).

In fact, I’ve talked before about reasons why A/B testing is always the right answer and examples of what an A/B test looks like. But, I’ve also been collecting a number of other posts that make the case for A/B testing. Check ’em out:

• I’ve talked about how to choose what to A/B test first before. But, I have to say, I love this look at 8 A/B testing case studies from Visual Website Optimizer’s blog.
• FOT (that’s Friend of Thinks) and guest poster Jake Stein of RJMetrics shows an A/B test he conducted for a landing page, complete with a full analysis of what worked best.
• Sometimes, even with lots of data, you’re forced to make a challenging call. GetElastic offers a case study that didn’t quite go as planned. But, even though the GetElastic study didn’t pan out as expected, take particular note of what the team learned.
• Evan Miller gives a quick overview of the math behind A/B testing that’s worth a read. Sure, you can just use calculators like the Teasley set to test your significance. And, admittedly, the math gets a little geeky, but if you’re serious about A/B testing, I’d take the time to check it out.
• Also, if you’re really into the math, Cennyd Bowles takes a deep look at statistical significance and other testing pitfalls. Heavy, but surprisingly readable. I’d also call attention to my comment on the post for a counter-argument to Cennyd’s concern.
• You don’t just have to A/B test your landing pages, of course. Jason over at A Smart Bear gives a great example (complete with cute hamsters!) of A/B testing on AdWords. Cool. (And, did I mention the HAMSTERS?!?)
• And, finally, if all that isn’t enough for you, Smashing Magazine has their Ultimate Guide To A/B Testing. And it is simply smashing.

Whew! That’s a ton of resources for you to use as you plan your own A/B tests. And you should.

Did I leave something out? I’d love to continue to collect these case studies here. If you find a good one, do me a favor and post it in the comments.

Are you getting enough value out of your small business website? Want to make sure your business makes the most of the local, mobile, social web? thinks helps you understand how to grow your business via the web, every day. Get more than just news. Get understanding. Add thinks to your feed reader today.

Or subscribe via email.

And while you’re at it, don’t forget to follow Tim on Twitter.

Technorati Tags:
a/b testing, adwords, Avinash Kaushik, call-to-action, click, continuous improvement, conversion, conversion rate, conversion rate optimization, E-commerce, e-commerce, ecommerce, getelastic, google adwords, internet marketing, landing page design, landing page optimization, marketing best practices, measurement, metrics, page design, testing, tools