When marketers like us create landing pages, write email copy, or design call-to-action buttons, it can be tempting to use our intuition to predict what will make people click and convert.
But basing marketing decisions off of a "feeling" can be pretty detrimental to results. Rather than relying on guesses or assumptions to make these decisions, you're much better off running conversion rate optimization (CRO) tests.
CRO testing can be valuable because different audiences behave, well, differently. Something that works for one company may not necessarily work for another. In fact, CRO experts hate the term "best practices" because it may not actually be the best practice for you.
But these tests can also be complex. If you're not careful, you could make incorrect assumptions about what people like and what makes them click -- decisions that could easily misinform other parts of your strategy.
One of the easier (and most common) types of CRO tests is called an A/B test. An A/B test simply tests one variable in a piece of marketing content against another, like a green call-to-action button versus a red one, to see which performs better. 
So, what does it take to run an A/B test, exactly? Keep reading to learn what an A/B test is in a little more detail, followed by a full checklist for what marketers should do before, during, and after these tests. You'll want to bookmark this for your next one.
How A/B Tests Work
To run an A/B test, you need to create two different versions of one piece of content with changes to a single variable. Then, you'll show these two versions to two similarly sized audiences, and analyze which one performed better.
For example, let's say you want to see if moving a certain call-to-action button to the top of your homepage instead of keeping it in the sidebar will improve its conversion rate.
To A/B test this change, you'd create another, alternative web page that reflected that CTA placement change. The existing design -- or the "control" -- is Version A. Version B is the "challenger."
Image Credit: ConversionXL
Then, you'd test these two versions by showing each of them to a predetermined percentage of site visitors. (To learn more about A/B testing, download our free introductory guide here.)
Now, let's walk through the checklist for setting up, running, and measuring an A/B test.
Checklist for Running an A/B Test
Before the A/B Test
1) Pick one variable to test.
As you optimize your web pages and emails, you might find there are a number of variables you want to test. But to evaluate how effective a change is, you'll want to isolate one, single variable and measure its performance -- otherwise, you can't be sure which one was responsible for changes in performance. You can test more than one variable for a single web page or email -- just be sure you're testing them one at a time.
Look at the various elements in your marketing resources and their possible alternatives for design, wording, and layout. Other things you might test include email subject lines, sender names, and different ways to personalize your emails.
Keep in mind that even simple changes, like changing the image in your email or the words on your call-to-action button, can drive big improvements. In fact, these sorts of changes are usually easier to measure than the bigger ones.
Note: There are some times when it makes more sense to test multiple variables rather than a single variable. This is a process called multivariate testing. If you're wondering whether you should run an A/B test versus a multivariate test, here's a helpful article from Optimizely that compares the two.
2) Choose your goal.
Although you'll measure a number of metrics for every one test, choose a primary metric to focus on -- before you run the test. In fact, do it before you even set up the second variation. If you wait until afterward to think about which metrics are important to you, what your goals are, and how the changes you're proposing might affect user behavior, then you might not set up the test in the most effective way.
3) Set up your "control" and your "challenger."
Set up your unaltered version of whatever you're testing as your "control." If you're testing a web page, this is the unaltered web page as it exists already. If you're testing a landing page, this would be the landing page design and copy you would normally use.
From there, build a variation, or a "challenger" -- the website, landing page, or email you’ll test against your control. For example, if you're wondering whether including a testimonial on a landing page would make a difference, set up your control page with no testimonials. Then, create your variation with a testimonial.
4) Split your sample groups equally and randomly.
For tests where you have more control over the audience -- like with emails -- you need to test with two or more audiences that are equal in order to have conclusive results.
How you do this will vary depending on the A/B testing tool you use. If you're a HubSpot Enterprise customer conducting an A/B test on an email, for example, HubSpot will automatically split traffic to your variations so that each variation gets a random sampling of visitors.
5) Determine your sample size (if applicable).
How you determine your sample size will also vary depending on your A/B testing tool, as well as the type of A/B test you're running.
If you're A/B testing an email, you'll probably want to send an A/B test to a smaller portion of your list to get statistically significant results. Eventually, you'll pick a winner and send the winning variation on to the rest of the list. (Read this blog post for a more detailed guide on calculating an email A/B test's sample size.)
If you're a HubSpot Enterprise customer, you'll have some help determining the size of your sample group using a slider. It'll let you do a 50/50 A/B test of any sample size -- although all other sample splits require a list of at least 1,000 recipients.
If you're testing something that doesn't have a finite audience, like a web page, then how long you keep your test running will directly affect your sample size. You'll need to let your test run long enough to obtain a substantial number of views, otherwise it'll be hard to tell whether there was a statistically significant difference between the two variations.
6) Decide how significant your results need to be.
Once you've picked your goal metric, think about how significant your results need to be to justify choosing one variation over another. Statistical significance is a super important part of A/B testing process that's often misunderstood. If you need a refresher on statistical significance from a marketing standpoint, I recommend reading this blog post.
The higher the percentage of your confidence level, the more sure you can be about your results. In most cases, you'll want a confidence level of 95% minimum -- preferably even 98% -- especially if it was a time-intensive experiment to set up. However, sometimes it might make sense to use a lower confidence rate if you don't need the test to be as stringent.
Matt Rheault, a senior software engineer at HubSpot, likes to think of statistical significance like placing a bet. What odds are you comfortable placing a bet on? Saying "I'm 80% sure this is the right design and I'm willing to bet everything on it" is similar to running an A/B test to 80% significance and then declaring a winner.
Rheault also says you’ll likely want a higher confidence threshold when testing for something that only slightly improves conversation rate. Why? Because random variance is more likely to play a bigger role.
"An example where we could feel safer lowering our confidence threshold is an experiment that will likely improve conversion rate by 10% or more, such as a redesigned hero section,” he explained. "The takeaway here is that the more radical the change, the less scientific we need to be process-wise. The more uber-specific the change (button color, micro copy, etc.), the more scientific we should be because the change is less likely to have a large and noticeable impact on conversion rate."
7) Make sure you're only running one test at a time on any campaign.
Testing more than one thing for a single campaign -- even if it's not on the same exact asset -- can do a number on your results. For example, if you A/B test an email campaign that directs to a landing page at the same time that you’re A/B testing that landing page ... how can you know which change caused the increase in leads?
During the A/B Test
8) Use an A/B testing tool.
To run an A/B test on your website or in an email, you'll need to use an A/B testing tool. If you're a HubSpot Enterprise customer, the HubSpot software has features that let you A/B test emails (learn how here), calls-to-action (learn how here), and landing pages (learn how here).
For non-HubSpot Enterprise customers, other options include Google Analytics' Experiments, which lets you A/B test up to 10 full versions of a single web page and compare their performance using a random sample of users.
9) Test both variations simultaneously.
Timing plays a significant role in your marketing campaign’s results, whether it's time of day, day of the week, or month of the year. If you were to run Version A during one month and Version B a month later, how would you know whether the performance change was caused by the different design or the different month?
When you run A/B tests, you'll need to run the two variations at the same time, otherwise you may be left second-guessing your results.
The only exception here is if you're testing timing itself, like finding the optimal times for sending out emails. This is a great thing to test because depending on what your business offers and who your subscribers are, the optimal time for subscriber engagement can vary significantly by industry and target market.
10) Run the test long enough to get substantial results.
Again, you'll want to make sure that you let your test run long enough in order to obtain a substantial sample size. Otherwise, it'll be hard to tell whether there was a statistically significant difference between the two variations.
How long is long enough? Depending on your company and how you execute the A/B test, getting statistically significant results could happen in hours ... or days ... or weeks. A big part of how long it takes to get statistically significant results is how much traffic you get -- so if your business doesn't get a lot of traffic to your website, then it'll take much longer for you to run an A/B test. In theory, you shouldn't restrict the time in which you're gathering results. (Read this blog post to learn more about sample size and timing.)
11) Ask for feedback from real users.
A/B testing has a lot to do with quantitative data ... but that won't necessarily help you understand why people take certain actions over others. While you're running your A/B test, why not collect qualitative feedback from real users?
One of the best ways to ask people for their opinions is through a survey or poll. You might add an exit survey on your site that asks visitors why they didn't click on a certain CTA, or one on your thank-you pages that asks visitors why they clicked a button or filled out a form.
You might find, for example, that a lot of people clicked on a call-to-action leading them to an ebook, but once they saw the price, they didn't convert. That kind of information will give you a lot of insight into why your users are behaving in certain ways.
After the A/B Test
12) Focus on your goal metric.
Again, although you'll be measuring multiple metrics, keep your focus on that primary goal metric when you do your analysis.
For example, if you tested two variations of an email and chose leads as your primary metric, don’t get caught up on open rate or clickthrough rate. You might see a high clickthrough rate and poor conversion rates, in which case you might end up choosing the variation that had a lower clickthrough rate in the end.
13) Measure the significance of your results using our A/B testing calculator.
Now that you've determined which variation performs the best, it's time to determine whether or not your results statistically significant. In other words, are they enough to justify a change?
To find out, you'll need to conduct a test of statistical significance. You could do that manually ... or you could just plug in the results from your experiment to our free A/B testing calculator. For each variation you tested, you'll be prompted to input the total number of tries, like emails sent or impressions seen. Then, enter the number of goals it completed -- generally you'll look at clicks, but this could also be other types of conversions.
The calculator will spit out the confidence level your data produces for the winning variation. Then, measure that number against the value you chose to determine statistical significance
14) Take action based on your results.
If one variation is statistically better than the other, then you have a winner. Complete your test by disabling the losing variation in your A/B testing tool.
If neither variation is statistically better, then you've just learned that the variable you tested didn't impact results, and you'll have to mark the test as inconclusive. In this case, stick with the original variation -- or run another test. You can use the failed data to help you figure out a new iteration on your new test.
While A/B tests help you impact results on a case-by-case basis, you can also apply the lessons you learn from each test and apply it to future efforts. For example, if you've conducted A/B tests in your email marketing and have repeatedly found that using numbers in email subject lines generates better clickthrough rates, then you might want to consider using that tactic in more of your emails.
15) Plan your next test.
The A/B test you just finished may have helped you discover a new way to make your marketing content more effective -- but don't stop there. There’s always room for more optimization.
You can even try conducting an A/B test on another feature of the same web page or email you just did a test on. For example, if you just tested a headline on a landing page, why not do a new test on body copy? Or color scheme? Or images? Always keep an eye out for opportunities to increase conversion rates and leads.
What else would you add to this checklist for running an A/B test? Share with us in the comments.
No comments:
Post a Comment