6 Split-Testing Mistakes You Should Avoid

These days, your business likely depends on consistent and ever-improving conversion rates through your website. Store visits aren’t enough. The problem is, how do we know if what we’re doing is working? Sales online go up one month, down the next, and up again after that. Is it the weather? A holiday? Did the government just open a new department?

You need to know what’s really working on your web pages, and what’s just coincidence. That’s where A/B split testing comes into play. There are whole companies that do testing, and software you can buy to do it yourself. But none of that does any good if you don’t test effectively.

What Is an A/B Split Test?

An A/B test takes a web page, called the control page (the A page), and changes one or more features on it. This new page is the variation page (the B page). These pages may both be linked to various AdWords ads, for instance, and data can be collected to see which pages produce more conversions.

It’s like science class, except now it’s your livelihood on the line, and not the frog’s.

So that you don’t end up like the frog, here are six A/B testing mistakes you do not want to make.

1. Don’t End Your Test Too Soon

One day is not enough time. Customers come and go during the week. Weekend habits differ from weekdays. Nights are not mornings. Holidays interfere. To get a valid test, control for as many factors as you can. The easiest one is time, and at least a week is a good minimum. However, some tests may require more time, others less. An AdWords ad split test might need less time than a home page test. If you use multivariate testing, it may be different again.

No matter which type of test it is, don’t end it too early. Getting statistically valid data (“significance” is the buzzword) is more important than meeting your boss’s artificial deadline.

Also consider the goal of the page. Is it a sales page, a free offer, or are you just measuring click-throughs? The length of time for the test may vary depending on this, in part because of the number of users visiting that page.

In fact, the number of users is another reason not to stop the test too early. You need enough users to get valid data. So if it takes two weeks to get a couple thousand users, but you feel a couple thousand is your minimum threshold, then keep the test going until you get there.

Otherwise, you may just have to do the whole thing over again anyway.

2. Don’t Add Unnecessary Variables

The most obvious variable under your control is when you run your tests. Be sure and run them simultaneously. You can’t do the A test one week and the B test the following week. What if Rihanna shaves her head and the internet goes berserk? This might affect your sales for a day.

But if you’re only running the A test that week, then you’ve got an uncontrolled variable skewing the data, and it’s essentially worthless.

But there are other variables. If you chance six things on the variation page, and your conversions go up, how do you know which feature made the difference? Or, maybe it was a combination of all six.

Now, this doesn’t mean you should run a test because you changed the font size of the headline, but it does mean that you must be aware of what you are trying to learn, and to avoid clouding the meaning of your output data with too many variables.

3. Avoid Testing Anxiety

Lots of students in school get nervous before tests, even though they know the material just fine. A/B test marketers ought to know better. Don’t be afraid of testing new ideas. It is very hard to predict how customers will react to your messaging. You will be surprised sometimes, guaranteed, because we aren’t all masters of human nature.

So if someone has an idea for a new way to do a page, then test it. What do you have to lose? A couple weeks with a dip in sales isn’t going to matter in the context of the whole year. And if the new idea causes an increase, then you’ve won, and learned something valuable you can try building on.

Keep testing ideas. It’s not like you’re printing any of this.

4. Usability is King.

Don’t neglect usability. Don’t neglect usability. Déjà vu, anyone?

There are a million things you can analyze with A/B split testing. Headlines, calls to action, your logo, your free offer, the font size – test them all. But keep in mind, users get easily annoyed when sites don’t work well. If they get hacked off when your video keeps playing automatically, or that no matter where they click, it takes them to the shopping cart page, they might just leave the page and write a scathing blog post about how lame your site is. Meanwhile, all your headline and logo testing schemes crash and burn into nothingness.

Make your site work right. Don’t neglect usability. And if you need help with it, hire a copywriter or a usability expert for consultation.

5. Don’t Stop Early

Make sure you test lots of stuff. If you try a new headline and see a 3% conversion increase, that’s great. But don’t stop testing and be happy with that.

What about the body copy? The offer? The shapes around the call to action button? The button copy? What about where on the page the copy starts? Or where you put your buttons? There are a ton of things to test. And all of them can make incremental differences that may add up to huge sales increases. That 3% might go up by 1, or 2, or 0.5 with each of these changes. Add it all up and your sales could go up 20%.

Never stop testing, and never stop improving the site. Besides, what worked last year might not work as well anymore.

6. Don’t scoff at small gains.

This isn’t the lottery. It’s business.

All those small gains add up to something sizable. So if you do a test, and the results show a statistically significant gain of 2%, don’t consider that nothing. That’s something you can build on. With those gains, test something else on top of that, and maybe it will go up to 2.5% from the original. Keep testing more changes, and the small increases will add up.

If you have a large business, even a 1% increase can mean a lot of sales.

Get On the Testing Bandwagon

Split testing has become an industry, because web marketing is an industry. A/B testing is one way we measure the effects of our web marketing. So if you’re not doing it at all, you should be. If you are, makes sure you avoid these six common mistakes, and you’ll be on the path to sales growth.

If you’ve made any of these mistakes in your previous testing efforts, how did you change your process to make your testing work better for you?