A/B testing! Does it sound like an experiment you would do in a Chemistry lab? If you have an online business, and little or no knowledge about A/B testing, this post is exactly what the chemist would prescribe for you.

This two part series will cover everything you need to know about A/B testing from the basics to knowing how to get started and what to test to ensure that you are getting the most out of your website, landing pages and other collaterals.

Let’s kick off the series with a post that will explain what A/B testing in detail, explain the difference between A/B test and multivariate testing and take a stock of what can be A/B tested.

The following post would help you get started with A/B testing and lay down the guidelines for effective A/B testing. You will also learn what to test and get your hands on some case studies. Be sure to stay tuned for the next part of the series!

Introduction to A/B testing

To some, science and art may look like oil and water- those just won’t go together isn’t it?

This debate is also what keeps the two teams in tussle when it comes to online marketing. There are some marketers who believe marketing to be more a form of art and others who feel it to be more of a science form.

left rightphoto credit: Foggy Hoarfrost Crosswalk Buttons via photopin (license)

The ‘art of marketing‘ focuses on harnessing emotions with visuals, content and story-telling.

The ‘science of marketing‘ focuses on data and analytics.

American author, entrepreneur, marketer, and public speaker, Seth Godin helped halt this debate by proposing marketing to be combined form of these two philosophies.

Just like the left side and the right side of the brain don’t work in separation, marketing too is both an art and a science, in equal measures.

If you focus too much on the science, your website and marketing will become robotic and your messages will focus more on conversion and call to action’ rather than providing value.

Likewise, focusing too much on the creativity without knowing what resonates with your viewers won’t help either.

The key is to strike a balance. After all, the purpose of the marketing is to make the company grow and so you need to have a balance between creativity and analytics.

Why is A/B testing important

Not sure if you should be A/B testing your website and landing pages just yet? Consider the following statistics:

  • Only 44% organizations leverage the use of split testing software which should be 100% – seogadget
  • Organizations who allocate more budgets for optimization have better success rate with their conversion – Marketing Charts
  • Less than 25% businesses were reported to be satisfied with their conversion rates – – Econsultancy
  • According to Pardot, in 2001 alone Google ran over 7,000 A/B tests.
  • According to Marketing Charts, almost 60% online marketers plan on doing customer journey analysis for improving their online conversion rates.
  • As per Conversion XL, only 1 out of 8 A/B tests drive significant changes.
  • Quick Sprout reported that Gmail decided on the highest converting shade after testing over 50 shades of blue for their CTA color.
  • According to Econsultancy, A/B testing is the most used method for improving conversion from the last two years.

What is A/B testing?

To take an analogy of litmus test, just like how you would check a solution to see if it is acidic or basic, similarly, A/B testing also called as bucket testing or split testing is a method to compare two versions of a web page with each other to determine their performance. Based on the analytics, A/B testing can be used for validating the page that would improve the conversion rate of your landing page.

A/B testing can help you run a variation of your web page, app or whatever it is that you are testing against the original. The data collected can help you understand the impact that the change has on the conversion rate. A/B testing takes out the guesswork and helps you make an informed decisions.

A/B testing can also help you run well-designed and carefully implemented experiment to evaluate which variant works best!

How to apply A/B testing?

From website copy, to landing page advertisements, emails, search ads, in fact, you can test anything that helps improve your website’s conversion rate.

Have you heard the story of potato, eggs and coffee beans? A frustrated daughter complained to her chef father about her difficult life. Her father took her to the kitchen and filled three pots with water while placing, eggs, potatoes and coffee beans in each.

Once cooked, he asked her to touch the three and explained that potatoes went in hard and came out soft, eggs went in soft and came out strong and coffee beans changed the water and created something new. All three of them faced the same adversity, boiling water, but they reacted completely differently to it.

Similarly in A/B testing, when you test different variations, you get different results, you just got to find the ones that react like coffee-beans.


Well, now you know what A/B testing is all about but it’s one thing to know something and completely another to apply it, isn’t it?

Take heed from arguably the most powerful man on Earth – President Barack Obama.

During President’s re-election in 2007, optimization was an integral element of the campaign. Overall, around 500 A/B tests were undertaken in a span of 20 months which helped improved the donation conversion by 49%  and sign up conversions by , wait for it – a whopping 161%.

Check out some of the screenshots to understand how various things affected the online campaign.

3By turning the long donation form into 4 smaller steps we increased the conversion rate by more than 5%.


Just a small change in the copy of the page resulted in an increased conversion rate of over 21%.

Source:  (http://kylerush.net/blog/optimization-at-the-obama-campaign-ab-testing/)


Just a small change of image resulted in an increase in conversion rate by over 19%.

We could go on and on offering you screenshots to show how A/B testing could help! But I guess you got the gist!

What benefits can a business gain from A/B tests?

It’s economical

It may seem like an extra cost since you are developing different versions but with the benefits that it offers, it is fairly minimal.

Most of the time, there are very small differences and it could usually just be split into halves and tested. After the test is completed and data has been gathered, it is easy to decide on which one performs better.

Besides, the end result is something that helps you increase your conversion rate which is any way beneficial.

You can accurately measure performance differences

With A/B testing, it is easier to measure performance difference, even if it may be a minor difference.

It helps you measures actual behavior

When you are developing something, you are not sure how it will be received by your target audience. With A/B testing, you can better judge on how whatever it is that you’re developing, will be received by your target market. This can also be helpful in long term by helping you create products and services that are more desirable. This jewellery website increased their conversion rate from 0.01 to 6% just by testing different variations of their landing page.

It can resolve trade offs

When you are working on your collateral, you are more often than not, worried about whether this color pattern will work, whether the wordings or strategy will bring you more conversion.

Easy to pivot

If your marketing is not being effective, it is easy to pivot and change to whatever works!

Ease of implementation

There are a lot of software packages available that support A/B testing. The data can be collected through web analytics tool and the changes can be implemented easily.

Ease of test design

Since, A/B testing are done usually with very few changes, it doesn’t take long to design the variation.

landing page styling 2 landing page styling 2 landing page styling 1

Ease of analysis

Only by comparing the baseline version and understanding the data is enough to analyze the change and take the desired actions.

Flexibility in defining the variable values

With A/B testing, you don’t need to restrict yourself to a granular level of variables as is the case with multivariate test (we will discuss multivariate test next).

How A/B testing works?

In A/B testing, you take whatever it is that you want it tested, and modify to create a variation of the same page with little changes. This change could be anything, from color, headline, button, number of form fields or alignment of call to action.

The original version of the page is known as control and the modified version of the page is called as the variation.

The visitors are divided and served with either the control or the variation and the engagement is measured through an analytics dashboard.

Based on the results, it could be determined whether the variation had a positive or negative or even no effect on the visitor’s behavior.


A/B test vs. Multivariate test?

A/B testing and multivariate testing are similar in the sense that both these types of testing are done to understand best variation the user resonates with.

While A/B testing is a to b testing that is one on one testing, as the name suggests multivariate is the kind of testing where multiple variables are tested. That is, in multivariate testing, various subtle changes are done to the original and all possible combinations are tested to determine which combination performs the best.

Multivariate is more complex form of testing than A/B testing as it helps to give you an idea on what elements of your web page or whatever it is that you are testing, interact with each other and offers you a combined result.

In multivariate testing, a few key sections of the page are identified and then different variations of those sections are created instead of creating variations for the whole page as done in A/B testing.

Multivariate tests could be used for refining and optimizing existing web pages without undertaking drastic changes. While in A/B testing, often radically different ideas are tested to understand and optimize conversion rate.

Traffic required for A/B testing is relatively less as the changes are radical and so the level of engagement could be derived easily.


Source: https://www.templaza.com/blog/images/multivariate.jpg


(Split Testing)

How to pick the right testing method?

Now, that you know what A/B testing and multivariate testing is, the next question you need to ask when you need to go for A/B testing and when should you go for Multivariate testing.

A.B testing is a great option when you are launching a campaign and when you need to measure the results fast. In case you need meaningful results to make quick decision, the best option for you is to go for A/B testing.

Since, the difference between the control and the variation of A/B testing method is big and noticeable, it is easier to understand which page resonates more with the users.

A/B is also a great method to choose especially if you don’t have much traffic on your website. For multivariate test, you need high traffics to get meaningful results.

Multivariate test is a good testing method when you want to understand how different elements of your page interact with each other to help you understand and improve on the current page.


When it comes to businesses, is it easier to get so focused on the daily grind so much so that we lose sight of important elements. Remember that just working on something is not enough, if you want to see the optimized results, you need to optimize your efforts too.

Besides offering you a higher conversion rate, A/B testing also brings you more closer to your customers. After all, you are using what resonates with them and that shows that you care about your customers beyond just the numbers.

Remember a proven commitment is better than a promised one.

Let us know your feedback and stay tuned for the continuing post in the series, A/B Testing – A complete Guide 2’ to get the best tips to improve your conversion rate through A/B testing.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *