What the Heck is an A/B – Split Test?

Simple, A/B testing is exactly what it sounds like:  you have two versions of an element (A and B) and a scale that defines success.  This scale is normally your click through or conversion rate.  To determine which is better, A or B, you would subject both to an experiment/test simultaneously.  In the end, you measure which version was more successful and select that version to move forward with.  The winning version in turn becomes your control element.

For instance, say you have two versions of an email (A and B) selling the same product.  Typically, A is the existing email that you know has converted well in the past (called your control), and B is the new version of the email.  You would send the email out to two equally segregated parts of your list and measure their performance using such metrics as: conversion rates, sales, bounce rate, etc.  In the end, you would choose the version that performs the best.

This is similar to the experiments that most of us did back in high school.  Although, in this case we don’t have to dissect a embalmed rodent or smell the chemical rotten egg smell!  Instead it’s all about bits and bytes, traffic numbers, and conversion rates.  The ultimate outcome resulting in an increased number of currency in your bank account.

What should you test?

What you choose to test will obviously depend on your own goals.  For example, if your goal is to increase the number of opt-ins, then you might test the following:  length of the opt-in form, types of fields on the form, which of the fields are optional and which are required, where and how big you display your privacy policy,  where the form is located on your webpage, colors, fonts, etc.  The goal of the split test in this case is to figure out what prevents visitors from opting in.  Is the form’s length too cumbersome or intimidating?  Are they concerned with privacy? Is the form appealing to the eye?  All of these questions can be answered one by one by testing the appropriate elements.

Even though every split test is unique, here are some elements that are predominantly used in most tests:

• Headline and subhead

• Layout and style of overall page design

• Call to action – button size, text, color, and placement

• Pricing and promotional offers

• Amount of text on the page (short vs. long)

• Video sales page vs. a traditional text page

• Offer flow – funnel

Your first split test

I bet you are asking, “Well, yeah this all sounds dandy, but how do I test?”  There are several tools available to help you with the more technical side of executing all your split testing needs.  If you are just getting your feet wet and desire a free basic tool, I would suggest Google Website Optimizer.  However, and this is a pretty big draw-back, Google Website Optimizer does require some monkeying with HTML and JavaScript.   So if you aren’t knowledgeable with either, you might want to go in another direction.   A few easier alternatives are:  Visual Website Optimizer, Optimizely, or Five Second Test.  Of course there are a ton of other options available, all ranging in a wide variety of difficulty and cost.   Although, setting up a core test is more or less similar for all the different tools, so let’s discuss the remaining tool-agnostic.

You can set up a A/B Split Test in one of two ways:

• Replace the element to be tested before the page loads to the viewer

If you are testing a single element on a page, let’s say the call to action button, then you will need to create variations of that button in your testing tool.  When the test goes live, the split test tool will randomly replace the original (control) button on the page with one of the variations before displaying it to the viewer.

• Redirect to another page (more widely used)

If you want to split test an entire page, let’s say the overall look and feel of the page itself, then you will need to create and upload a new page on your website.  Example:  if your homepage is http://www.example.com/index.html (control), then you will need to obvsiouly create a variation located anywhere on your site.  For example the variation might be placed here:  http://www.example.com/index1.html (variable).  When your test runs, your testing tool will then redirect some of the traffic to one of your alternate URLs.

As soon as the conversion event occurs on your site (click of the button, purchase, etc), the split testing tool records the results.  After a sufficient amount of traffic and conversions have taken place, you can check the results and find out which variation resulted in the most conversions.  Then switch out the winner to become your new control.

How long should you run the test for?

Split testing typically is not an overnight project.   Depending on the amount of traffic you are sending to your site, you might want to run the tests anywhere from a few days to a couple of weeks.  Ideally you will want to run one test at a time for the most accurate results.

If you don’t give your test a sufficient amount of time, your results could be skewed.  Alternately, running a test for too long can also give you a different kind of skewed results, since there are more variables at play that you can’t control over a longer period of time.  If at anytime though you are in doubt, it’s perfectly reasonable to retest.

Considering the impact split testing can have on your bottom line, though, it’s worth taking a week or more to properly conduct the tests.  Test one variable at a time, and give each sufficient time to play out.

Can you test more than one thing at a time?

The simple answer to this is yes!  Let’s say that you just want to test your headline, but with three possible variations.  The most effective way to do this would be to run a single test and split your traffic into three groups.  This is more efficient than running three separate tests (A vs B, B vs C, and A vs C).  You would definitely want to give your test an extra couple of days though to run, so that you have enough results to base your conclusion on.  This kind of test is called a multivariate test, and is more complicated to run than a simple A to B.   However, still doable even for a beginner.

Split testing can increase your bottom line

Accurate split testing can make a huge difference to your overall profit.  By using controlled tests and gathering observed data, you can figure out exactly which marketing strategies work best for your site and/or product.

Split testing done consistently, can improve your bottom line substantially. If you know what works and what doesn’t,  and have the evidence to back it up, it makes decisions that much more simple.  You can then craft more effective marketing materials to get you closer to your overall goal.  Remember though, your testing is never done.  The market changes quickly and often, so it’s important to keep up the testing.