Table of Contents
A/B testing. Somehow you know you need it. But have you ever thought about the actual reasons it is an absolute marketing strategy game changer?
The obvious ones are that it boosts your subscriber opt-in rate, improves conversions and skyrockets your online sales.
And we’ve only scratched the surface.
Did you know that one out of eight a/b tests has driven significant change for an organization?
If this sounds like something your business could make use of, keep reading.
What to test, how to test it, and how to analyze your results. It’s all here. And it will help you get better results, guaranteed.
What is A/B Testing?
A/B Testing, also known as Split Testing, is an unemotional, entirely logical, coldly scientific way to compare variables in your marketing side by side, to determine which of the two variables works best.
The History of A/B Testing
The first person to successfully use A/B testing was William Sealy Gosset, who, in 1908, wanted to determine which strain of barley produced the best ale for Guinness (yay, beer!) in the over 100 years since, A/B testing has been used for everything from direct mail marketing to engineering.
But in the age of internet marketing, A/B testing has really come of age, with companies large and small tweaking the minutiae of their online presence to squeeze every drop of revenue possible from their efforts.
Because A/B testing is cheap. It’s fast.
And it’s increasingly easy to test and refine websites, apps, and email marketing messages — at least it’s faster, cheaper, and easier to modify web pages and email than it is to modify the ingredients in barrels of beer.
How does it work?
Let’s say you have a landing page with a sales offer that is currently getting 10 clicks a day. You want to see if you can improve that. You’ll use your current page as “A” — the control.
Then you’ll tweak ONE variable on that landing page — the headline, the color, or any other ONE single variable — and create a version “B” of that page.
You’ll then display the A version to half your website visitors and the B version to the other half over a specific time period — a day, a week, a month — or as long as it takes to generate a statistically relevant number of clicks.
At the end of that experiment, you’ll have cold, hard data about which version of your landing page produces more clicks — the “A” version or the “B” version.
Let’s say that the B version generates 12 clicks a day. Obviously, that represents a 20% improvement in the click-through rate. Once you know that “B” works better, you’ll use B as your Control, and move on to the next element to test.
One more example
Or let’s say you are marketing to a mailing list of 1000 people, and you want to improve your open rate.
Use your original email message as your control:
Version A. Tweak ONE variable (subject line, sender, greeting, or message) and use that message as
Version B. Send half your subscribers the Version A email, and half of them Version B.
Examine your open rates, and you’ll see the winning version — A or B. Then move on and test the next variable.
Numbers don’t lie.
Why A/B Testing?
In 2019, industry experts estimate that 1.92 billion people will buy something online. And mobile e-commerce will generate something on the order of $2.32 trillion in sales.
Want some more interesting A/B testing statistics?
According toEntrepreneur, “better UX design as a result of user testing can increase a company’s conversion rate by 400%“.
Whether you’re new to online marketing or a seasoned pro — whether you’re a small company or a giant retailer — A/B testing is the fastest, easiest, and most cost-effective way to discover how to drive more traffic, generate more leads, create more sales, and claim a bigger, smarter share of the $2.32 trillion pie.
A/B testing can give you a massive competitive advantage.
Odds are your competitors aren’t even doing A/B testing — or if they are, they’re not doing it right.
That leaves you plenty of room to learn, grow, and create content that converts like crazy in your marketplace.
So smart marketers “Always Be Testing.” If you are — or want to be — one of them, read on for everything you ever wanted to know about A/B testing.
A/B Testing Checklist
Let’s revisit the basics of online marketing to answer that question.
All online marketing starts with traffic. You have to do something (advertise) to drive traffic to your website or landing page.
Are you getting all the traffic you can?
Run A/B tests on your ads. Test your headline, copy, image, call to action, and any other element of your ad to ramp up the amount of traffic you’re getting.
Once visitors land on your page, you need to motivate them to willingly give you their contact information.
Most marketers offer some sort of lead magnet, or “ethical bribe.”
Are you getting all the opt-ins you can?
Run A/B tests on your lead magnet, landing page, contact form, sales process, and other elements of your offer to find out.
Testing each variable will help you improve your opt-in rate because you’ll know which variable works best.
How are your open rates, click-throughs, sales?
Run A/B tests on your email campaigns.
Examine your subject line, greeting, sender, and the message itself to make sure.
Rather than frantically testing everything at once, start your testing from a rational, scientific, data-driven hypothesis.
A/B Testing Analytics
To generate a working hypothesis, you have to know your numbers. To create meaningful A/B tests, at the minimum you should track these essentials:
- Traffic: total visitors, unique visitors, repeat visitors
- Traffic Source: search, ads, referrals, social media
- Bounce rate: the amount of time visitors spend on your website before clicking away
- Conversion Rate: the number of visitors taking the action you want them to take (opting in, buying, downloading, etc.)
- Cost per lead: the cost of your campaign divided by the total number of conversions
- Lead to Close Ratio: the number of sales divided by the number of leads
10 Tips for A/B Testing Success
When you’re first starting out, the whole notion of A/B testing might get very confusing and very complicated very quickly — especially if think you have to correct everything at once.
Use these “best practices” to keep your testing clean and your results clear.
1. Test ONE hypothesis at a time
Let’s say, for example, you want to improve your conversion rate in an email campaign.
Your email directs people to the landing page for your product.
Let’s further say that you want to test both a segment of your audience and an image on your landing page to see which one converts better.
If you try to run both tests at the same time, you won’t know whether it’s the particular audience or the particular landing page image that affects your conversions.
2. Set up Control (A) and Treatment (B)
In any scientific experiment, you have to start with a “Control” — the original version of the element you’re testing (A).
From the Control version, you’ll systematically build variations or “Treatment” (B) versions of the landing page or email you want to test.
3. Determine how “statistically significant” your results should be before running your A/B test
What kind of statistical significance would make you decide to change your landing page or email campaign?
You need to know in advance what results would drive your decision to change.
If you aim at 95-99% statistical significance, you can be 95-99% sure that the difference in performance isn’t due to chance, and you can trust your decision to make the changes.
4. Test the entire landing page or email first
Before you drive yourself crazy testing a button, a color, an image, or a headline, try testing your entire landing page or email instead of testing every minor design or copy element.
Just design two different landing pages or write two different emails and test them against each other.
It’s almost a certainty that one version will outperform the other, and once you’ve identified the winner, you can start testing individual variables.
5. Test ONE variable at a time
Once you have a winning page or email, you can drill down and test the individual elements of design and copy.
Just make sure that you test only ONE variable at a time, otherwise, you’ll muddy your results.
6. Test as far down into your funnel as you can
As you proceed through various A/B tests, you will improve your conversion rates.
But don’t make conversion rates the be-all and end-all of your testing.
Look at your other metrics — how is your A/B test impacting traffic, click-throughs, leads, or sales?
In some cases, you might find that even though a version of your page or email converts fewer prospects, it generates more sales.
You’ll never know if you don’t examine the deeper elements of your funnel. (Hubspot and Kissmetrics have great tools to help you analyze your results.)
7. Test the “invisible” elements of your page or email as well as the obvious ones
When you’re looking at A/B testing, it’s easy to think about the minutiae of the visible — color, layout, font size, column placement, headline, buttons, long-versus-short copy, and so on.
But don’t shy away from testing the things that aren’t visually apparent like different target audiences, promotion timing (time of day, day of week, month, season), or alignment between an email and a landing page.
8. Test the small stuff, too
You don’t have to limit your A/B testing to major elements like headline, copy, or images.
Sometimes the tiniest changes can make the biggest difference.
Google, for example, tested over 41 different shades of blue before deciding on the most effective shade for their famous toolbar!
9. Test equal, random audiences
To make sure your results are conclusive, test two or more “equal” audiences.
For example, if you’re A/B testing an email campaign, each version of your test email needs to be sent to as similar an audience as possible.
Some factors that might cause huge differences in response rates include list source, list type, and the length of time people have been on the list.
That’s why you’ll want to make sure to split your lists randomly, keeping all the other elements of the design and timing identical.
That will get you the most conclusive results.
10. Test simultaneously
Meaningful A/B testing requires that you operate like a scientist — and that includes running your tests at the same time.
If you were to run test A one month, and test B the next, you won’t be able to determine the real cause of the different response rate. It could be as simple as the time of year.
So, whether it’s the time of day, a day of the week, or month of the year, run your tests simultaneously to eliminate time as a variable.
Now let’s look at how to apply A/B testing to the two most important elements of your online marketing efforts: your landing page or website, and your email campaigns.
How to A/B Test Your Website
There are over 1.95 BILLION websites in the world, as of January 2019, according to Internet Live Stats, with more than 300 new sites going live every single minute.
If you’re a website owner, you owe it to yourself to do whatever you can to get your voice heard in all that noise and maximize your web presence.
As your traffic online grows, even a small percentage increase in web traffic can make a massive difference to your bottom line.
Take, for example, Spreadshirt, an online marketplace where users can create and upload designs they can have printed on t-shirts, bags, and other accessories, and sell through the Spreadshirt online marketplace.
Under the leadership of a new, testing-savvy creative director, they completely revamped their website.
One of their first hypotheses was that they might increase engagement and sales by simplifying their home page, specifically their “Start Selling” call to action.
A simple test — making the call to action simpler and more prominent on the page — created a mind-blowing 606% increase in clicks and a 4% increase in orders.
Could you use a boost of over 600% in your website clicks? If so, start testing!
6 Components that Benefit from A/B Testing
It’s possible to find all kinds of research online about what works best in online marketing.
Hubspot is full of marketing statistics if you want general information or guidance.
But while general statistics like “long-form blog posts generate 9 times more leads than short-form” or “organic traffic improves 157% with videos” might suggest that you immediately write longer copy or add video to your site, keep in mind that those statistics are general.
They might not apply to your business or your audience.
Your audience is unique to you.
Your audience might like video…or they might hate it.
Your audience might want to see a 200-word bullet-pointed blog post they can skim…or they might want a 5000-word post they can curl up on the couch with.
The only way you’re going to be able to provide the kind of online experience to your target market is yearning for is to test, test, and test.
Even minor design changes can drastically influence customer behavior, as Google found in its “41 Shades of Blue” experiment.
While A/B testing every detail of design might seem like a huge waste of time and energy, even the tiniest improvement in click-through rates can represent more money as your website traffic grows.
Think about it this way: Visuals play a huge part in a customer’s decision to engage or buy.
As a matter of fact, JLB Media reports that 9 in 10 shoppers say visuals are the most important factor in their buying decisions.
So, the way your website or landing page looks will make all the difference in how your visitors experience and engage with it.
Here are some of the design elements you might want to test on your website or landing page and here are some landing page examples:
- 1, 2, 3 columns or more
- Navigation links and location – top versus left column versus right column
- Background color
- Font colors
- Navigation link/toolbar colors
- Featured image
- Header image
- Background image
- Rotating images
Your website visitors have come to you for a specific answer to a specific problem.
Their needs should drive your content.
For example, if your target audience includes senior-level executives, a short, bullet-point article with links might work better than a half-hour video.
If, on the other hand, your audience is highly technical, they will probably respond better to longer copy with plenty of data and statistics.
Tailor your content to how much your audience already knows, how much they need to be educated, and how interested they are, according to your own specific marketing plan.
Test these elements:
- Short-form vs long-form copy
- Video vs Audio vs Text vs Slide Show
3. Contact Form
Don’t make it difficult for your visitor to opt-in by asking for too much information. A general rule of thumb for contact forms is “less is more.”
But consider testing:
- Number of fields
- Text link vs button to “submit”
- Location on the page
4. Call to Action
What, specifically, do you want your visitor to do? How much information do you need to provide them so they’re ready to take that action? Your call to action needs to be straightforward.
But do test:
- Position – “above the fold” vs “below the fold”
Test the best option for pricing your product or service.
Will you offer a free trial? Will you offer a one-time-only gift with purchase? Should you offer “decoy” or limited time only pricing?
- Free trial
- Length of trial
- “Decoy” pricing
6. Steps to Complete Order
Put yourself in your website visitor’s place and examine your entire funnel process from his or her point of view.
Make it as easy and seamless as you can for them to do business with you.
Jumio reports that “While the majority of smartphone and tablet owners (68%) have attempted to make a purchase on their device, two-thirds (66%) failed to complete a transaction due to obstacles encountered during checkout.”
And StreetInsider further estimates that “Cart abandonment accounts for $24 billion in sales losses each year.”
You will want to test:
- Number of steps to order
- Shipping costs
- Barrier to qualify for free shipping
- Payment methods
As you can see, A/B testing your landing page or website could be a challenging process. Setting up the test, managing the details, and gathering and analyzing the data might seem overwhelming.
It doesn’t have to be. There are plenty of great website A/B testing tools with price tags ranging from free to tens of thousands of dollars.
Depending on the size of your company, your budget, your staff, and the time you have available, the recommendations below can help you streamline the whole A/B testing process.
5 Great A/B Testing Tools for Websites
1. Google Analytics
Google Analytics forms the baseline for any data analysis on the internet.
You can connect your website to Google Analytics, and start collecting invaluable data about its performance within minutes.
You can set and track conversion goals and set up quick, efficient A/B tests that Google calls “Experiments.”
You’ll be able to easily find your website’s bounce rate and analyze statistics that show you how visitors are behaving and interacting with your site.
The good news is it’s free to anybody with a website.
The bad news is you’ll face a steep learning curve as you will have to interpret the raw data yourself. Luckily, it’s fairly intuitive to get up and running.
If you’ve never done data analysis yourself and want to start A/B testing your site, Optimizely is one of the best choices you can make.
It’s simple to install, has a low-cost entry point, and features a fantastic, highly-intuitive user interface.
You can also add features and functionality as your budget grows.
Optimizely features a specialized tool that allows you to take two (or more) variations of pages on your website and compare them side-by-side to see which performs better.
With just this tool, you can easily increase visitor engagement and conversions, and make numerous changes to your landing pages without having to dig around in your code.
3. Visual Website Optimizer (VWO)
A little pricier than Optimizely, VWO is gaining in popularity as a full-featured suite of testing tools aimed at improving conversions.
Like Optimizely, they offer a very simple, intuitive interface, but feature a whole lot more options that you can use as little or as much as you like.
Their platform is based on a logical, 5-step system: Track, Analyze, Plan, Test, and Target.
Users are systematically guided through the process by the interface. One of the features that make VWO so popular is its highly intuitive funnel analysis system that allows you to track data deep within your own funnel and see where you might be losing leads.
This tool shows you where you need to focus your A/B conversion testing first.
The testing dashboard is visual, so you can make any changes to your landing page easily by selecting and changing the element you want to test within seconds.
When you’re finished, VWO will create all the possible combinations of those changes and help you choose the best option.
That way you can run a full round of testing on one single element –without having to change it manually every single time.
Unbounce has cornered a very specific niche in the A/B testing world.
They only work with landing pages. That’s it.
Well, that and simple, basic A/B testing. And they’ve mastered the art of creating and testing high-converting landing pages.
What that means is that Unbounce is a very basic, tightly focused tool.
You can literally create and publish conversion-optimized landing pages in just a few hours, using Unbounce’s more than 80 pre-tested templates that you can customize for your brand.
If you’re just beginning your online marketing adventure and are looking for fast results, this is a great tool to create and start A/B testing your landing pages at a very reasonable cost.
Perhaps one of the most robust, data-driven, insightful testing software on the internet, Kissmetrics has been around a while.
Its A/B tester allows you to create experiments and evaluate the results so you can achieve conversion goals throughout the depth of your website.
Almost as robust as Google Analytics, Kissmetrics is much simpler and more intuitive to use. It’s also one of the more expensive options.
You can start by A/B testing conversions from page views, and then drill down through subscriptions and sales, exploring and improving every element of your sales funnel.
But one of the standout features of Kissmetrics is their ability to store the historical details of every single visitor to your website, tracking their behavior, and giving you insight into who they are and what they want from your site.
Knowing who they are and what they want allows you to create custom populations for your website testing.
So now you’ve got a fully tested, high functioning landing page that’s getting tons of opt-ins. It’s time for the follow-up email.
And guess what?
You’re going to have to test that, too.
How to A/B Test Your Email Campaigns
According to the Radicati Group, over 3.8 BILLION emails are sent every day, with the expectation that by 2022, 333 BILLION emails will zip around the internet.
And according to the same Group, the average office worker gets something approaching 121 emails every day (while only sending 40).
Nearly half of those billions of emails (49.7%) is considered spam.
Given that nearly half of the email out there is considered spam, and 2.3% of the worldwide email contains some sort of malicious attachment, is it any wonder that the average open rate for marketing emails (in North America) is 13.7% on mobiles, and 18% on desktops.
What’s a marketer to do?
Easy. Improve your open and click-through rates by A/B testing every email campaign you send.
A/B testing will help you identify and address the needs particular to your audience, and what works best in talking to them.
The more you test, the more you’ll be able to uniquely position your business to address those specific needs.
Over time, you’ll be able to create custom-tailored email campaigns that precisely target your prospects and generate much more engagement.
In other words, A/B tests will save you time and money as you identify the precise elements that help your business succeed.
But again, just like with your landing page, you need to know your numbers before leaping into A/B testing, so you’ll know what you’re measuring against.
For email A/B testing, you need to know:
- Open rate
- Click-through rate
- Response Rate
- Subsequent Conversion Rate
The email campaign software or platform you use should help you analyze those numbers.
Email A/B Testing Best Practices
- Begin with a specific goal in mind. Do you want to improve your open rate? Your click-throughs? Your response rates? Or your conversions down the line? Testing is meaningless without measurement.
- Use a large enough sample to make sure your results are statistically significant.
- Randomize your sample groups.
- Test before you launch the entire campaign, so you’ll have time to analyze and interpret your results or make any changes before you send it to your entire list.
- Test often — test every campaign.
- Identify each variable you want to study and test them one at a time.
- First, consider testing two completely different emails against each other to get a feel for the kind of content your audience prefers. Then drill down and test the variables.
The 4 Email Campaign Elements You Should Test
1. Subject Line
Typically, you only have 30 characters to catch your reader’s eye with your subject line. The optimal length should be between 60-70 characters. Finding the right subject line through testing could make the difference between 200 people or 2000 people opening your email.
The ONLY purpose of your subject line is to get your reader to open the email. Some other subject line considerations to test include:
- Which tone works best, a neutral and businesslike, friendly and intimate, or provocative and shocking?
- Does rearranging the words in my subject line make a difference in my open rates?
- Will a long or short subject line work best for my subscribers?
- Does including a number in the subject line improve my open rates?
- Does using the recipient’s name in the subject line work for my audience? (On mobile devices emails with a personalized subject line are 25% more likely to be opened. Your mileage may vary.)
- Will my subject line work better as a question or as a statement?
2. Campaign Content
If the purpose of the subject line is to get your readers to open the email, your first line of copy should give them a reason to keep reading and engaging with your content.
You’ll want to test components of your copy content including:
- Should I use long-form copy or short copy?
- Should I use an image in my email? Which performs better, a message without a visual, or the same message with a visual? Should I use one visual element or many?
- What colors create better open rates?
- Does the font type, size, or color influence my open rates?
- How does the way my content is spaced impact open rates?
- Does one email template outperform another?
- Does a button work as the call to action to link to my landing page? Or does a text link work better?
3. Campaign Sender
Depending on your product or service, the sender of your email might influence your open rates.
Would your audience prefer to get a message from, for example, Bill Gates? Or from Microsoft? From Jeff Bezos? Or from Amazon. Elon Musk? Or SpaceX? Test it.
- Should I send the email as a person?
- Should I send the email as a company?
The internet is crawling with studies about “the best time to send emails.”
But you won’t know for certain whether those general recommendations apply to your company and your audience.
The only way to discover when your subscribers are free from distraction and more likely to open your emails is to test.
Once you uncover your best timing through A/B testing you’ll be better able to motivate them and get them to take action.
You’re going to want to test the best:
- Time of day
- Day of week
- Time of month
- Season of year/holidays
6 Great Email A/B Testing Tools
Just like testing your landing page or website, AB testing your email campaigns might seem complex and confusing.
It doesn’t have to be. There are lots of email campaign services available, and, luckily, most of them offer A/B testing as part of their plan.
Here are just a few recommendations:
Moosend has everything you need to easily automate — and A/B test — your email marketing campaigns, whether it’s a newsletter, a customer onboarding letter, or a drip-feed series.
Moosend’s platform features effortless and flawless email design, user-friendly list management, state-of-the-art automation, and reliable analytics.
You can automate on the go, design your own automation workflows, and sync up your data without restrictions.
Instead of offering a limited-time free trial, Moosend offers all its robust email services free — forever — as long as your list is fewer than 1000 subscribers.
As your list grows, you’ll pay a bit more, either by a monthly fee starting at $10 for more than 1000 subscribers.
You can also “pay as you go,” if your list size changes.
A full-service email and contact management service, AWeber features simple, easy-to-use tools to create email campaigns and related opt-in forms to manage all aspects of email marketing.
Starting from a 30-day free trial (during which your emails will contain an AWeber watermark), AWeber’s pricing starts at $19 per month for up to 500 subscribers and ranges up to $149 per month for up to 25,000 subscribers. A/B testing, or Split Testing is just one feature of their service.
Infusionsoft is a monster of a contact and business management platform that allows you to organize your sales, marketing, and customer information all in one place.
Of the tools available for email campaign management, it’s one of the most expensive.
For established businesses with sophisticated marketing needs, prices start at $199/month for list sizes up to 2500 subscribers, and range in excess of $399/month and up as your subscriber base grows.
With a simple, intuitive drag-and-drop interface, you can create marketing campaigns using their ready-made templates.
You can also build landing pages from optimized, industry-specific templates and A/B test everything to improve conversions.
GetResponse offers more than just email campaign management.
Calling itself an “All-In-One” marketing platform, GetResponse provides tools and resources for not only email management but also landing pages, webinars, audience behavior tracking and more.
Offering a free 30-day trial (with restrictions), prices start at $15 per month for email marketing and landing page tools for lists up to 1000 for one user account.
Prices range up to $1199 per month full-service plan for lists up to 100,000 subscribers and 10 user accounts.
GetResponse allows A/B testing on emails, ads, webinars, landing pages, and all funnel elements in between.
5. Constant Contact
Like most other contact management/email marketing software, Constant Contact begins with a two-month free trial and offers tiered pricing for a full-service email-management-only platform.
While plans start at $20 per month, the least expensive way to get A/B testing is by spending $45 per month (depending on your list size, it could be more).
Constant Contact does, offer, however, customizable email templates optimized for mobile email, as well as all the metrics, analysis, and reporting you might need.
MailChimp is a robust online marketing platform that allows you to start marketing from ground zero.
It offers you the ability to create landing pages, popups, and Facebook and Instagram ads, with more services available as your subscriber base increases.
You can use the basics of Mail Chimp free for up to 2000 subscribers, but your emails will be sent with MailChimp’s logo and branded text.
Tiered pricing starts at $10 per month and ranges upwards of $199 per month, based on your list size.
A/B testing is included in the free plan, so if you’re just getting started at email marketing, Mail Chimp offers a reasonable entry price point — free — if you’re willing to send emails out with their logo and branded text.
Here are also some Mailchimp alternatives to consider.
A/B Testing Examples
It might not seem much changed. But, as Design for Founders reports, “a simpler design plus a stronger CTA led to 433% more clickthroughs“.
The green button, the decluttered page and the changed copy of the CTA did their magic apparently.
Here’s one of the simplest a/b testing examples I came across, but effective too.
Whil the copy and structure of the Arono‘s landing page basically remains the same, a different image makes all the difference. In fact, the first example converted 53% better than the second.
We all know how important your free content is for your business. A
nd getting your visitors to download it is easier than you may think. It might just even be a matter of color.
Like this button here that by becoming much more visible gained 28.7% more conversions.
More information usually means more conversions. Kivi, a non-profit organization, got 11.5% more donations just by adding a F.A.Q. and providing statistics.
That concludes our massive A/B testing breakdown!
Hope you found some resourceful tricks and tips to optimize your next A/B Testing efforts.