How many times has your designer kicked out a ‘final’ creative asset based on best practices, gut feelings, “experience”, or if you’re lucky, historical data? Ever get the sense they nailed it on the first try? I haven’t and it can be so frustrating to ask for multiple versions or styles. Well, when resources are constrained, the best thing to do is utilize your knowledge of A/B and Multivariate testing.
Testing Methods
There are several methods to employ when testing components of your creative assets, but the two most common are; A/B and Multivariate testing. For the sake of simplicity we will focus on these two and leave the more complex testing methods to the professionals.
A/B Testing
A/B testing is defined as the method of testing a baseline Control [original creative asset] compared to a single-variable Sample [your revised asset] to improve response rates. Making sure you only have one variable being tested during the process is critical to success. Skewing outcomes can be very easy when introducing multiple iterations if you’re not controlling for them.
A simple example, if you’re A/B testing a subject line in an email (Control and Sample) to see which has the higher open rate, keep it as simple as possible. Send out the emails at the same time, don’t change email headlines or CTA buttons and only test for subject line. If you introduce other variables, you will skew your results. Alternatively, if you’re testing the color of a call-to-action button, you will want to keep everything static on the page, only test one variable at a time; your call-to-action button.
I like having several versions of an asset queued up to test. Use the existing asset in place as your Control and have several different Samples in the queue ready to rotate and test. Test your Control to Sample 1, if the Control wins, test the Control against Sample 2. If Sample 2 wins, test Sample 2 against Sample 3, etc. I call this simple A/B progression testing.
Controlling for variables outside of your website or design is important. For example, control for; large social events (the 2008 Olympics skewed tests I ran during that time), time of day, day of week/month, and most importantly, any seasonality your business experiences throughout the year.
Multivariate Testing
When looking to test more than one variable at once, you’ll want to engagement the methods of multivariate testing. Defined as the process by which more than one component of a website (or creative asset) may be tested in a live environment. For example, perhaps you have five different hero images and five different headlines you’re interested in seeing which combination yields the highest click-through rate. Ensure you have the right technology to randomize the 5 different hero images and headlines evenly across time to ensure a large enough sample size interacted with your test before drawing conclusions.
Another form of testing multiple variables is what I like to call multilevel-multivariate testing. An email example of this includes; testing different subject lines, the color of your email’s call-to-action button, and the copy of your Sign Up page’s “submit” button. This form of testing involves three assets and flows through three different interaction levels; 1) opening the email (subject line), 2) clicking-through from email to website (call-to-action button) and 3) conversions on your website’s landing page (submit button).
Testing Sample Size
I can’t tell you how many times I’ve rolled out a simple test and got so excited when I saw my Sample was winning. I would run around and tell everyone the new design was winning and yell and scream we should roll it out 100%. Well, thankfully there were others there to tell me to shut up, calm down and check my sample count. You can’t make rash decisions based on premature data and sometimes need an impartial party to keep you in check.
There are a few components you need to understand prior to testing; sample size, confidence level, confidence interval and population.
Sample Size:
- These are the users exposed to your test, the more users within your sample, the greater the likelihood your results will represent your entire population.
Confidence Level:
- This is a percentage telling how accurate your test is. It symbolizes how likely your entire population will behave as your sample population behaved. Most tests are done until they achieve a 95% Confidence Level, meaning you’re 95% certain that your sample population reflects the same behavior as would your entire population.
Confidence Interval:
- This is an interval representing the percentage difference from your sample results. If your confidence interval is ‘+/-5’ and 60% of your sample population responded positively to your test, then 55% to 65% of your entire population will respond the same.
Population:
- This is the number of total users in your entire population. For example, if your email list is 1MM people, then your ‘entire population’ would be 1MM users, your sample size may only need to be 5,000 to hit the confidence interval you’re looking for. You’ll derive the needed sample size from your population size and desired confidence interval.
Now that you understand the fundamentals of your needed sample size, prior to launching your test you should have an agreed upon confidence interval. Typically, a 95% confidence interval is an 
acceptable margin of error. Depending on the value of your test, you may be inclined to accept a lower confidence interval or if you’re testing a multi-million dollar outcome, you may want to see a 99% confidence interval.
Lastly, to ensure your sample users best represent your entire population, make sure they are randomized. Again, this oversight will skew results and not reflect the opinions or behaviors of your entire population. Here is a free confidence interval calculator to determine the sample size for a corresponding confidence interval and population size.
You’re now set, ready and excited to kick off your next creative asset test. For additional information on testing creative assets, refer to Testing Creative Assets for Stellar Performance. For all of your affiliate marketing needs, there is HasOffers; the leader in Affiliate Tracking Software. Please feel free to contact me or leave comments below
Nice post. I think the best testing strategies are those that incorporate all of these aspects and keep the testing going, even if one of the creatives seems like a clear cut winner. It’s easy to see what’s working and what’s not, but it’s difficult to say “this is the best combination of creatives” so there is literally no end to the testing you can do. Thanks for the post.