Testing Silver Bullets

A/B testing

When creating a website for the first time, it’s common to turn to well-known experts in the field to get their magical solutions or “silver bullets” on how to improve it.  

Thanks to the possibilities of A/B testing, I decided to experiment by performing multiple tests on some of their advice. And to my surprise, the results were different from predicted.  I guess the experts weren't wrong, just that testing shows that in the business world, silver bullets don't exist.

The experiment I'm going to tell you about here was very simple.  Should the action on a landing page be at the top or the bottom?  The conventional wisdom of my close advisers was, unanimously, that it should be at the top.

To test this silver bullet, we designed an experiment where we ran the variations simultaneously for 40 weeks after first restarting the statistics.  The action with which we measured the conversion was the page display vs click on the "I WANT TO BE HEALTHY" button. (Here shown in green, “QUIERO SER SALUDABLE”). What does the experiment tell us about this silver bullet?:

In the table above we see three variants: A, AA and B. Variants A and AA have the button at the top of the page:

Variant B has the button at the bottom of the page:

As we can see in the first image, the conversion rate of the A and AA variants is between 6.10% and 7.63% (average 6.86%). However, the B variant has a conversion rate of 13.70%, a 79.55% improvement over the best previous conversion.

The difference between variants A and AA followed another, more subtle suggestion, where I was told I should change the order of the sections:

Again, the change proved to be insubstantial and contrary to the recommendation. The difference in conversion rate between Variant AA and Variant A was marginal (1.53%), that is, practically statistical equality. Andin any case the original Variant A, where “investment” appeared before “experience”, proved to be better.

The simple conclusion is that if someone gives you a silver bullet solution, you have to test it out.  In the long run the circumstances surrounding a business context are usually different and the important thing is to make identifiable changes one at a time, measure, compare, go back and repeat.