How much testing is too much testing?

How much testing is too much?

When are designers right? When are engineers? How do you know how much testing is too much? Designer Douglas Bowman announced his decision to leave Google last week, largely for this reason:

“Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle.”

Shortly after Bowan’s announcement, such diverse outlets as Jeffrey Zeldman, Joe Clark and CNNMoney’s Jon Fortt questioned whether Google’s process is killing design (or as Jeffrey Veffer calls it “Design”). Clark goes so far as to suggest Google smothers designers.

Nonsense. Ignore this noise. While “form vs. function” or “design vs. engineering” makes for a great “religious debate,” ultimately it’s pointless. Great designers can help you be successful. So can great engineers. And when they work well together, truly amazing things can happen. But, not testing the work of either can lead you to failure. Falling into the trap of either/or is a poor decision. Making this an “x vs. y” ignores the power of great design AND testing.

Y’see, the flaw in this gnashing of teeth around the web is that the folks jumping all over Google on this one seem to assume only one path leads to success. Either you have a great designer or you’re doomed to failure. Many people point to Steve Jobs’ passion for design fueling Apple’s turnaround – and rightfully so. But, despite its (supposed) lack of design savvy, Google seems to be doing pretty well for itself despite how few design gurus are throwing parades in its honor.

Jeffrey Veffer points out,

“…this is not design (or “Design”) that is the equivalent to window dressing or chrome or garish menus, but really a holistic approach to approaching problems… [Bill Buxton] mentioned that design is probably the “most negative” profession out there as it required a continual progression from a blank sheet of paper (millions of possibilities) to exactly one, with all the rest being thrown out!” [emphasis mine]

Isn’t A/B testing one way of throwing out possibilities? And isn’t A/B testing within the reach of most businesses?

Think about this. The number of conversion actions you need to see to have a statistically significant sample for a given improvement looks like this:

(Source: Tim Ash, “Landing Page Optimization”)

So, sure, testing 41 shades of blue is overkill. For you. But not necessarily for Google. Given Google’s 140 million unique visitors monthly, about how long do you think it would take Google to test 41 shades of blue and get 100,000 samples each? Not long at all. And given that 0.63% of Google’s Q4 revenues equals about $36 million, it might make sense for them.

Who knows? Maybe with the additional revenue Google makes from this test, they can afford to hire more designers.

My point here isn’t to defend Google. They can defend themselves. Nor is it to criticize Bowman; he can – and should – work for whomever he chooses, using whatever methodologies suit him.

The point is that testing works. I’ve tested major changes – loved by designers – that had little effect. I’ve tested minor tweaks – despised by engineers – that drove big lifts in revenues. But we only knew what worked and what didn’t because we tested. Companies that work towards an optimization culture get the best results from both their designers AND their engineers. Anything less is a sucker’s bet.

Disagree? Think designers are nuts and engineers know best? Or vice versa? Don’t be shy. Tell me what you think in the comments below.

Are you getting enough value out of your small business website? Want to make sure your business makes the most of the local, mobile, social web? thinks helps you understand how to grow your business via the web, every day. Get more than just news. Get understanding. Add thinks to your feed reader today.

Or subscribe via email.

And while you’re at it, don’t forget to follow me on Twitter.

Image credit: Scott Schram via Flickr using Creative Commons Attribution 2.0 Generic.

Technorati Tags:
a/b testing, analytics, bryan eisenberg, call-to-action, Google, how to measure anything, internet business, landing page design, landing page optimization, measurement, metrics, online marketing, page design, solutions, strategy, success