Why “negative” test results are not necessarily negative.

A/B testing is a very humbling task. It teaches us that getting a negative result does not necessarily mean you have failed, but rather that there is room for improvement. When you A/B test you will quickly realize that most of your ideas might not originally be winners. However, don’t let this discourage you. An A/B test will always teach you something new, something on which you can iterate. In this customer case featuring Revolution Race we will show you how we went from seeing no uplift or change in behavior at all, to seeing an uplift towards finished purchases.

Revolution Race is a Swedish outdoor apparel company founded in 2014. They aim towards selling high quality outdoor clothing for an affordable price.

In the beginning of our project with Revolution Race we noticed that they have quite a unique size selector on mobile. First off there is in general a big focus on size and product fit. So even before you get to choosing your size you are informed on the details so that you can make an informed decision on which size you need. On product pages the sizes are toggled out by clicking on the CTA button, they then appear in a pop-up which sits along the bottom of the screen.

In addition to the size selector we are also met by a descriptive text about the fit of the product, as a way of helping the user understand what size might suit them.

But is having the size selector in a pop-up and hidden from the default view of the product page really the way to go? Could we, by presenting this information directly on the page, improve on the flow of the funnel and increase conversions?

Test #1

What we did for our A/B test on this was that we simply took the size selector and displayed this above the CTA. We also placed the descriptive text for size fit above this.

So, I’ve already told you what the results of this test showed. After running the test for 30 days we ended up with results which told us that the changes in the variant made no difference on conversions. We even saw a slight decrease towards adding to cart. However, we did not stop there. Working iteratively is a crucial part of conversion optimization. If you aren’t learning something from a test you’re doing it wrong. You need to put on your thinking hat and start digging into the why. What could be the reasons for the variant not outperforming the original?

Test #2

Now, you might think that having detailed information about size and fit would be beneficial at this step. By presenting this information before the size selector we allow for the user to make an educated decision on which size to pick, right? However there could be a possibility that this also makes the user question the purchase even more. There’s all of a sudden a lot more information that we have to take into account before deciding on which size to pick which makes us uncertain. It also pushes the CTA quite far down on the product page. Based on this we then tried a variant in which we scaled down and minimized the execution further.

As you can see in this variant we have taken out the information on size fit, it is instead displayed further down on the page along with the reviews. This is based on a hypothesis of how we think users behave when they are uncertain about the fit, that they then move more towards reading reviews to make sure that they are picking the right size. So by showing these aspects earlier on in the funnel we are instead interrupting the flow of decision making.

After running this variant against the original execution of the product pages for around 20 days we got the following results. For Add to cart we saw an indication towards a negative effect. However at the same time we saw an uplift of just below 4% towards Finished purchases, statistically assured at 99%. So even though we can’t see a larger difference in users who add products to their cart, we can see that the changes in the variant to a higher degree motivates towards finalized purchases. Based on this we can then draw a conclusion where the added information on size fit, at this stage, indeed could have been a distraction which meant that the uncertainty around the decision on purchasing increased and users are less likely to then complete their purchase.

So when you encounter a test result which is not showing the uplifts you expected, which those who A/B test a lot knows happens more often than not, don’t give up. You just have to dig a little deeper.

Do you want to learn how you can use A/B testing to increase conversion?