Are you currently utilizing split tests in your marketing efforts?
Your answer better be yes, and if not, you should probably stop reading this article immediately and start implementing split tests right now.
If you ARE currently split testing, are you using a statistical significance calculator to interpret your results before taking action?
Again, your answer better be yes, or you very well could be wasting your time with split testing and losing out on a lot of profit.
Split testing and not correctly interpreting and taking action on those results is without a doubt the number one biggest mistake I see affiliates making. Either the affiliate isn’t split testing at all, or the affiliate is making unfounded assumptions about the results.
I’ve been in the industry since 1999, as an advertiser, affiliate, and have owned multiple CPA networks. I can literally count on two hands the number of affiliates I know that correctly utilize split testing and calculate the results properly. This is such a HUGE problem and almost nobody pays attention to it.
For those of you that aren’t familiar with statistical significance, to strip it down to a brief summary: A statistically significant result is one that is unlikely to have occurred by chance.
So why should you care about this as an affiliate?
Statistical significance sounds like some nerd-driven term that you should completely ignore. Big mistake.
Probably the best way to illustrate how important it is for you to understand this concept is to look at some split test results.
Here are some actual results from a targeted grant campaign I was running. This is the first group of 100 visitors:
Offer 1 – 100 visitors, 3 conversions = 3% conversion rate
Offer 2 – 100 visitors, 8 conversions = 8% conversion rate
After 100 visitors each, offer 2 has more than double the number of conversions and conversion rate of offer 1.
Which offer would you run? Well with more than double the conversions after 100 visitors it’s pretty clear offer 2 is the hands down winner right? …Wrong!
I’ll spare you from the math formulas, as there are a number of solid statistical significance calculators out there that will do the math for you. However, with this particular test, here are some interesting mathematical results…
For offer 1, we can be 95% confident the actual conversion rate long-term will fall between .19% and 5.81%.
For offer 2, we can be 95% confident the actual conversion rate long-term will fall between 3.52% and 12.48%.
Those are some HUGE differences. So assuming that offer 2 is the winner from this split test could cost you a LOT of money long term. In fact, here are the results after 5000 visitors:
Offer 1 – 5000 visitors, 290 conversions, 5.8% conversion rate
Offer 2 – 5000 visitors, 237 conversions, 4.7% conversion rate
Offer 1 was the true winner and converted 1% better. If I would have just eye-balled the split test results like most affiliates do, I would have chosen the wrong offer long term, and for just this one campaign the difference in profits would have been over $75,000 in the first year alone.
I don’t know about you, but I hate missing out on $75K extra per year just because I didn’t use a statistical significance calculator to make sure my split test decisions were correct.
As an advertiser I can’t count how many times I’ve dealt with an affiliate promoting one of my offers that calls me or sends me an email mid-day that says something like, “I am pausing the offer, and I want to know why my conversion rate for the day so far is half of what it normally is.” This of course is after like 500 visitors and he of course has not done any statistical significance calculation at all.
I normally do my best to explain to them the offer hasn’t been touched so it’s something on their side, the ad network side, the cpa network, or much more likely…it’s simple statistical deviation.
As you can probably imagine, that usually flies like a ton of bricks.
Quite honestly it’s hard to blame them because the numbers can look so very blatant and obvious in one direction when in fact it might be the opposite.
I shouldn’t really need to drive this point home any longer…lets get straight to the tool you need to start making better decisions about your data.
The guys over at visual website optimizer also have a handy little web-based calculator which can be found here:
The web-based tool doesn’t quite give you as much information and detail as the downloadable calculator, but it’s great for a quick check.
If you’d like more information on how to use the calculator in detail, be sure to checkout the video on this page where I do a complete walk through for you, as well as provide some extra insight into statistical significance and how you can leverage it to make more money.