Multi variant testing and the flying squirrel

At university I was lucky enough to be taught by one of the leading thinkers in Evolution, John Maynard-Smith. I didn't really appreciate it at the time but of all the lectures I attended back then, his have stuck with me.

One of the biggest changes that has hit our industry in the last few years is multi variant testing or MVT. Sometimes called A/B testing MVT allows a site owner to try lots of alternate designs and to measure which performs best. This allows site owners to make small incremental changes that gradually improve the design - design evolution.

John Maynard-Smith spoke with enthusiasm about a concept called convergent evolution. He spoke about 2 animals. The flying squirrel and the soaring eagle. Both animals had evolved to fly but in very different ways. The flying squirrel jumps and sort of glides between branches. In the air for seconds, never gaining height. The soaring eagle on the other hand can reach heights of 15,000 feet and stay in the air for hours.

Herein lies the lesson for MVT.

Fly like a squirrel

The squirrel evolved bit by bit to fly. Small incremental improvements that allowed him to jump further, land better, and eventually to glide. The squirrel has reached a point where he isn't going to get better at flying (called a local maximum). The famous 41 shades of blue experiment by Google where they tried variants of button colours to find an optimum is a classic flying squirrel. Small tweaks equal flying squirrels.

Soar like an eagle

Of course everybody wants to be the soaring eagle. Elegantly drifting through the air with little effort. The eagle also evolves, little by little to its own maxim. It's a superb flyer, soaring to 15,000 feet instead of making long leaps between trees. The eagle started from a better set of basics: front limbs to wings, reptile scales to feathers.

Good design gives a good starting point for evolution. MVT can get you to a maxim, but it can't help you choose the best starting point to get to the highest maxim. A strong design leads to elegant flight. A weak design leads to hopping between branches. We need to combine MVT with craft.

It's about art and science

Here's some suggestions from what we've seen work and not work:

1. Just because it hasn't worked in one context doesn't mean it won't in another. I've been in meetings where I've heard "we can't make the header [colour]. We MVTed it and it led to a fall in conversion" Sure enough in the context of flying squirrel some things won't work (a squirrel with feathers is never going to be any better at flying) but in other contexts that design change will work.
2. Don't test each and every detail. This is becoming an increasing problem. Each small design change is made, MVTed over a 6 month period. Leading to analysis paralysis over MVT results. Too much data. After this some Frankenstein style page results. There is no craft here only more flying squirrels. Doug Bowman, a former Google designer talks about his frustrations at this approach.
3. Redesign as a whole, complete flow. User test, get some qualitative 'why' reasoning behind the design problem. MVT only gives a small insight whereas a even the most basic of user tests gives much richer results to work with. Design is about the holistic not the minutiae.

Remember, MVT is not a panacea for great design. Craft something before evolving.
Soar high, don't settle for being that flying squirrel.