3 reasons why A/B testing is always the right answer (Guide to Small Business E-Commerce Strategy)
Do you A/B test your site’s content, commerce and customer experience? You should, no matter how big your site. Especially in a down economy, the worst decision you can make is to abandon testing. If anything, now is the perfect time to look for ways to improve. And testing is the best way to find the improvements that work best.
But what if you site gets little traffic? Just the other day someone told me, “My site doesn’t get enough traffic to bother with testing. Why should I wait six weeks – or more – to get data when I need business results now?”
Fair question. Here are 3 reasons why testing matters, no matter the size of your traffic:
- Time isn’t the issue. Conversion rate is. If your test case performs dramatically better than your original, you’ll probably know it (relatively) quickly. You can feel comfortable a 15% increase in conversion wasn’t just a fluke in as few as 200-250 conversions[*]. Said another way, you only need so many conversions to prove what works. The bigger the increase in conversion, the fewer conversions you’ll need to know it’s working. And the bigger the increase in conversion, the less time it takes to collect the data.
- Meaningful changes take less time to show up. If you don’t see a significant lift in a timely period, it may mean the new version isn’t working dramatically better than the original. If that’s the case, the amount of time it takes to collect the data isn’t the problem; the lack of lift is. Put your focus on the areas where you’ll see greatest improvement. Not sure where to start? Check out the tips here, here and here.
- Sometimes the cure is worse than the disease. What happens if your changed page actually performs worse than the original? You could lose sales and not know whether it was the new version that hurt you or some external factor that would have impacted sales in either case. If you don’t run the two versions side-by-side, how do you know which works better?
Yes, you’ll still need to run your tests for a bit of time. Anything less than a couple of weeks worth of data could be influenced by many factors, including day of week, time of day, holidays, what have you. But don’t focus on that. Focus on what you’ll learn and the results you’ll see.
Simply put, the right answer is never, “don’t test.”
* – This is, admittedly, a rough figure that assumes an 85% confidence level. To check statistical significance for your tests, use Google Website Optimizer’s built-in tool or download the Teasley Stat Calculator [Excel spreadsheet] here.
Are you getting enough value out of your small business website? Want to make sure you’re ready for the mobile, social, local web? Subscribe to thinks today and get more than just news. Get understanding.
This Post Has 14 Comments
Leave a Reply
You must be logged in to post a comment.
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Tim, I couldn’t agree more that testing is the right way to gather data on improving website performance. I checked out the articles that you linked to, and they provided a good overview on how to choose the pages on which you should prioritize your testing.
I’m curious how you think about prioritizing types of media within individual pages. When you have the option of altering marketing copy, “buy now” links, positioning of contact details, product tour movies, etc, how do you decide where to start?
Jake,
I have the perfect answer for this. But I’m going to make you wait a day or two for it, because it’s such a great question, it deserves an equally good answer. Give me a couple days, and I will write it up the answer as a separate post. Thanks for the comment and keep me honest on the follow-up.
Good to hear, Tim. I’ll look forward to reading the post.
[…] between converting a sale and seeing your site visitors vanish. And, as we looked at last week, nothing works better than A/B testing to find out what your customers will respond to. But, Jake Stein asked a great question: How do you […]
Jake,
It took me a couple days, but here’s the answer you’ve been looking for. Let me know what you think.
[…] Vox says so. Of course, you know what I suggest. A/B test to see for yourself what’s best for your business. A 10% lift is nothing to sneeze at. But […]
[…] testing? Whether it’s headlines or copy, images or information, calls-to-action or content, A/B testing is always the right idea. Maybe you’re wondering what to test […]
[…] So can great engineers. And when they work well together, truly amazing things can happen. But, not testing the work of either can lead you to failure. Falling into the trap of either/or is a poor decision. Making this an “x vs. y” […]
[…] not impossible. By following best practices for developing your key performance indicators, testing where possible and segmenting your keywords into appropriate ad groups, you can attribute more […]
[…] know A/B testing is always the right answer, right? Why? Well, here’s everything you need to know. Whether you call it kaizen, continuous […]
[…] working on the right layout to show all this additional product information, don’t forget to A/B test your designs. Of course, while you’re at it, you also want to avoid Search Engine Land’s 6 deadly […]
[…] 3 Reasons Why A/B Testing Is Always the Right Answer […]
[…] too. Pick any one of the items on their latest landing page makeover and try it on your site. Or A/B test it against your current site for the best […]
[…] fact, I’ve talked before about reasons why A/B testing is always the right answer and examples of what an A/B test looks like. But, I’ve also been collecting a number of other […]