A/B testing: To be or not to be

Marketing Performance Marketing Strategies Marketing for Beginners

No pain, no gain and no leads. To turn the traffic your ads bring into sales, you need to constantly optimize your site by improving the user experience, changing user behavior, and increasing your conversion rate. But how can you make sure the changes you implement will bring about the expected results? That’s what A/B testing is for. In the article, we tell you what A/B testing is, how to carry it out, and what nuances are worth paying attention to.

Book a demo

Gain clarity for better decisions without chaos

No switching between platforms. Get the reports you need to focus on campaign optimization

Book a demo

What is A/B testing?

A/B testing in marketing is the same as split testing — a comparison of two variants of a website page that differ by just one parameter. The goal of A/B testing is to determine which of these two options is more effective and brings more conversions.

Let’s say you sell software. You have a landing page with a product description and a button at the bottom of the page to subscribe to a trial version. To increase the number of subscribers, you decide to add one more button on the landing page for those who already know about your product or prefer to try it right away without reading the details.

To check if your hypothesis is correct — that you’ll get more subscribers by adding another button — you create a copy of the original landing page and add a button to it. Then you divide page visitors into two groups: one that will be shown the original page (variant A), the other that will be shown the updated page (variant B). At the end of the test, you compare the performance indicators (in our example, the number of subscriptions) and determine the winner.

A/B testing

Why conduct A/B tests?

Let’s look at some of the main things A/B tests (or split tests) help us achieve. 

1. Better understand your users and give them what they want. No matter how long you’ve been doing ecommerce and online marketing, it would be a mistake to rely solely on your personal experience.

Even if it seems you can predict the behavior of website users and understand exactly how to organize the content so they move as quickly as possible through the funnel of sales, conduct an A/B test. The results may surprise you. 

As practice shows, our assumptions don’t always coincide with reality. Therefore, we can’t decide what’s best for our customers based only on our own beliefs. 

2. Rely on data over expert opinion. A second challenge, which arises from the first, is the feasibility of making changes on the site and minimizing the risks involved.

Often, hypotheses are based on personal views that may not coincide with audience views. As a result, changes introduced without A/B testing don’t have the desired effect — or worse, they reduce conversions.

Therefore, when you’re faced with the question of what to use for decision-making — data or expert opinion — always choose data.

3. Personalize communication with customers. Customers use different devices, come from different sources, interact differently with your site, browse and buy different goods...

Web analytics services such as Google Analytics help you combine this data and systematize knowledge about users. Marketers collect information about what pages users visited and what they did on them. This makes it possible to divide the audience into several tens or hundreds of segments and to learn, for example, how users who come from organic or paid traffic behave.

But we don’t always use this information correctly and don’t always squeeze the maximum benefit out of it. A simple example: most online projects still show the same content to all users, regardless of their behavior and traffic source:

Content personalization

If you’re doing that, a split test can help you fix the situation and personalize the content on your site.

Main stages of A/B testing

Now let’s take a look at the main stages (and nuances) of split testing:

Stages of the testing

Stage 1. Identify the problem

The first thing you need to do is identify a list of weaknesses on your site. To do this, you can:

  • Explore data in Google Analytics and other web analytics systems to see which pages have high failure rates, low scrolling depth, and poor conversion rates.
  • Use Webvisor and click heat maps to understand how users interact with elements of your site.
  • Analyze support cases or interview active customers to see what they’re missing on the site. 

For example, say you look at the Enhanced Ecommerce funnel in Google Analytics and see that very few people add a certain item to the shopping cart. At the same time, you have an offline point of sale and you know that this product is popular. In this case, most likely something is wrong with your online store.

Stage 2. Put forward a hypothesis

Once you’ve decided what to fix, you need to think about exactly how you’ll fix it. Without an A/B hypothesis, testing makes no sense — the value from your findings will be small. You should clearly understand the purpose of the experiment, what element of the web page you will test, and what quantitative results you want to achieve.

When formulating the hypothesis, push back against your conversion funnel. Ask yourself: What should I change in one part of the page to move the user through the funnel faster?" Check one hypothesis per test; otherwise, it will be difficult to define what change affected the end result to what degree.

What can be tested:

  • Color, size, text, and location of conversion buttons
  • Headings — change the text; make it shorter, more interesting, and more relevant
  • Forms — reduce the number of fields or add tooltips and fill examples
  • Landing — change the page structure, font, or color palette
  • Content — add high-quality photos and video, appeals to action, promotional offers, the word “free,” etc.

Your choice of performance indicators depends on your hypothesis and the goals you want to achieve. These can be revenue, number of purchases, conversion rate, average check size, applications and subscriptions, failure rate, etc.

Stage 3. Check metrics

The next step is to make sure you have all the necessary metrics implemented and recorded, on the basis of which you will draw conclusions at the end of the test. In our work, we’ve encountered cases where clients have identified weaknesses and formed hypotheses but have not correctly prescribed a system of metrics so they can understand that the conversion rate has changed precisely because of a change to a button, for instance, and not because of other factors.

Stage 4. Run the A/B test

Consider the following factors before running your experiment:

  • Minimum sample size. To ensure your test results are statistically significant and can be trusted, determine the required number of participants. You can do this with free online calculators such as Abtasty and Optimizely. Let's say the conversion rate of your original landing page is 5%, and you expect that the test version of the page will reach 7%. The minimum visible effect in this case will be 40%. Enter these numbers into a calculator and you’ll see that you need a minimum of 1964 people per variant:
Calculator of the sample size
  • External factors: seasonality, holidays, shares, weather, currency exchange rate, etc. So external factors don’t distort the results of the experiment, it’s important to show both versions of the page in parallel during the same period.
  • Test macro conversions first. If you set a goal to visit a certain page, it’s likely that users will achieve it but will not make a transaction or will not take another target action. It’s always necessary to think about your funnel as a whole to understand which user actions on the site are of the highest priority.
  • Consider the type of device. If you start an experiment on all traffic to your website and you have mobile and desktop versions, check how the test option looks on mobile devices.
  • Exclude internal traffic so your employees’ actions on the website don’t distort the statistics. This can be done in Google Analytics by means of IP address filtering.

After you’ve considered these factors, you can run the test. A little later, we’ll tell you about the tools you can use to do this.

Stage 5. Analyze the results

At the end of the experiment, analyze the results. For example, say the original conversion rate on your landing page was 3%, you assumed that you could increase it to 5%, and the test variant showed 3.5%. The conversion rate has increased, but just slightly. Now you need to decide whether to introduce the change on the site or try another hypothesis. 

You can check whether the results of the split test have statistical significance by using an online calculator or statistical methods.

Read more about statistical power, sample length, confidence intervals, statistical significance, and how to measure them in our article on statistics in web analytics, or how to become a true data scientist.
If the process is successful and you’ve received reliable data, bring the landing page winner to the site and proceed with the next experiment.

Possible errors when analyzing the results: 

  • Prematurely evaluating results. We recommend to carry out a split test for at least 14 days. You can make an exception to this rule if the task is on, you’re testing minor changes that don’t affect global functionality of the site (for example, you’ve changed the color of a button), and you’re using Google Optimize. If you see in your Optimize report that the new option wins with a probability of 80—90%, you can stop the experiment. The indicators are unlikely to change dramatically.
  • Evaluating results at a validity threshold of less than 95% is another metric from Optimize reports. When you conduct an experiment, Google Optimize considers the validity of the final result. If it’s below 95%, Optimize will recommend continuing the experiment. You can see this threshold in the tab with an active experiment.
  • Ignoring test results as minor. Who doesn’t want to double conversions all at once?! Even such a modest (at first glance) conversion rate increase of 2—3% isn’t a bad result, however. Especially if the changes on the landing page were small.
  • Not checking the global indicators for your site. After all, you need to check your global site indicators, not just the ones you’ve chosen as part of the experiment. A single parameter may not be enough to evaluate the effect of changes. For example, the average check size may decrease and the total revenue may increase by increasing the conversion rate. Therefore, monitor all interconnected KPIs.

Tools for A/B testing

To run an A/B test, you must create a test version of the page, segment your audience, and calculate the target metrics for each segment separately. If you have programming skills and enough resources, you can run an A/B test manually. But it’s easier and more convenient to do it with the help of special tools. 

We’ve prepared a small table comparing popular split testing tools:

testing tools

At OWOX BI, we use Google Optimize for tests, so we’ll focus more on the features of this tool.

A/B testing with Google Optimize

Optimize is an online service that’s connected to your website and allows you to experiment with different ways of displaying content.

Google Optimize

Optimize allows you to use the data you’ve accumulated in Google Analytics to offer a user the version of the page that will be most convenient for them and most profitable for your business.

Advantages of Google Optimize

  • Completeness of data. To set up and analyze an experiment, it’s possible to use purposes and segments from Google Analytics. You can work with usual metrics from Google Analytics that you know and love.
  • Ample opportunities for personalisation. After successfully completing a test, you can configure a demonstration of different content using Google Analytics audiences and variables that are implemented, for example, in dataLayer in Google Tag Manager. If experiments allow you to improve the productivity of your website for the average user, then personalization based on information about users will allow you to achieve higher returns within each segment.
  • Integration with other Google products for deeper targeting and analysis (Google Ads, Data Studio, Tag Manager, etc.)
  • A convenient interface that’s easy to understand. The visual editor allows you to configure and start new experiments without the involvement of developers. It significantly reduces the time for carrying out an experiment.
  • Minimally affects page loading speed.
  • It isn’t necessary to manually summarize data, prepare reports, and apply statistical formulas to check the results. Google Optimize does everything itself.

Optimize shortcomings

  • So far, Google Optimize cannot be used to test mobile applications.

  • You can’t schedule tests. That is, if you want to prepare tens of tests but can’t start them at the same time for some reason — or if there are restrictions in the free version on the number of simultaneous tests, or you don’t want to try out tens of options on the same audience, that might become a problem. You'll need to launch each test manually in the interface. This isn’t a critical shortcoming, but nevertheless you can do this in some other services.

How Google Optimize works

Google Optimize works similarly to other tools for conducting experiments and personalization:

How Google Optimize works
  1. First, you need to create variations of pages, pop-ups, and other objects that you’ll show to the user.
  2. Then you need to determine the goals (metrics) by which you’ll determine the winning option. These can be the metrics built into Optimize — number of page views, session duration, transactions, revenue and failure rate — or any custom goal from Google Analytics.
  3. After that, you need to identify the audience that will participate in the experiment and launch the experiment. At this stage, you must decide how much risk you can take by showing the test option to users. You can distribute traffic between two options equally or, for example, do a 20/80 split. Plus, at this stage, you have to choose which part of the audience you’ll show the experiment to. Show everyone, or take 20% and distribute your two options between them? Why might you want to do this? In case you have a large store, you’re unsure of your hypothesis, and you don’t want to risk half the traffic. 

In addition to the classic A/B tests, in Optimize you can run multivariate tests (in which you have multiple changing elements in multiple combinations) and redirected ones (for pages with different URLs and designs).

You can learn more about the interface and settings of Google Optimize in our article on how to conduct your first A/B test with Google Optimize.

Analyze the results

With reports in Google Optimize, you can monitor the results during your experiment and analyze the collected data immediately after it ends.

Google Optimize report

Terms in Optimize reports:

  • Improvement — the likely range for the conversion rate
  • Probability to be best — the probability that this option is better than all others
  • Probability to beat baseline — the probability that this option will bring a conversion rate better than the original
  • Conversion rate — the predicted average conversion rate
  • Conversions — the number of sessions with conversions

How the winner is determined
Google Optimize uses Bayesian inference to generate statistics. Without going into the details, this means that during the experiment, you can see in the Optimize reports the probability that variant B will be the winner before the end of the experiment. If the probability reaches a certain level, it’s possible to finish the experiment ahead of schedule and save time and money. 

In addition, the Google team plans to implement a mechanism for traffic redistribution in favor of the best option before the end of the experiment. This will save you money, as fewer users will see an ineffective option during the test.

If you integrate Optimize with your Google Analytics account, you’ll be able to browse and analyze the results of tests in the Google Analytics interface in the Behavior / Experiments section:

Behavior / Experiments section

If your experiment was successful, you can deploy the winning option on your website.

Links to useful materials