A/B Testing Step by Step [Complete Guide]

What’s more effective on my landing page? A red or a blue “Buy now!” button? Oh, wait! Maybe I should write “Add to cart” instead? The menu – would it look better on the left or right? Which picture speaks better to my customers? These kinds of questions are being asked everyday thousands of times. But there’s another one: Who’s the one answering? Usually it’s a project manager, programmer, graphic designer or Ms. X from Y Department. They’ve got their opinions and there’s not much we can do about it. Ask three designers about an idea, and they'll probably come up with three different landing page templates each presenting a different vision – but that’s perfectly normal and fine. So who’s right? The answer’s pretty clear: not a project manager, not a designer nor Ms. X, but the users. Your audience is diverse, remember that. Some will eagerly use a red “Buy now!” button, while others will most likely avoid any buttons with exclamation marks on them. Conversions on your landing page depend significantly on what users do and how they react to particular elements on your landing page: headlines, images, copy or the buttons mentioned above. So if you really care about creating a savvy and lucrative landing page, you’ve got to run some A/B Tests. Let’s start from the very beginning.

Table of contents

• What is A/B Testing?
• A Man With a Plan
• How to maintain A/B tests?
• A short guide to prepare a test campaign
• 5 things you should remember while A/B split testing

What is A/B Testing?

"A/B Testing is a marketing technique where two different versions of a landing page (website, email, etc.) are created and then released simultaneously to compare the results", says Miłosz Krasinski, Managing Director at a web consulting company Chillifruit.com The landing page should be tested by sending the first version to half of your target audience and a second to the other half. Then set an ending to the test (for example after 100 conversions, 500 visits or two weeks). Be sure to gather enough data for A/B testing. There’s not much to be learned about which variant is better if there were only 25 conversions.

Source: AB Testing, Author: Seobility, License: CC BY-SA 4.0.

It’s there for a reason, of course. The website was created to explain what Landingi is all about, and the video does it in a quick and accurate way. Positioning the video lower on the website would be pointless, because we want it to be seen. However, sometimes the video is just an add-on and placing it in the first section of a landing page could be distracting.

A Man With a Plan

In general, every process works better with a plan – it’s common knowledge. Of course, there are some exceptions to this rule, but split testing is not one of them. If you want to predict the results somehow, you’ll need to divide the process into a few stages. Lucky for you, there’s already a scheme for split tests:

1. comparing specific elements of a website
2. measuring results (mainly conversion rate) for individual variants
3. choosing the best combination

Simply put, you need to choose what to test and start testing. After some time/conversions/visits, check for results and choose which variant works better. Next steps are:

4. landing page optimization (implementing changes) 5. further monitoring of the conversion rate 6. another A/B test (optional)

Actually there’s no strict ending to the A/B testing. Your landing page can get better with every iteration, but if you plan to test it continuously, ask yourself a question: how long will you use this particular landing page? Here’s an example: at Landingi, we have been using the features landing page (mentioned above) for about a year or so, and we will use it until someone comes up with a better solution. This is why we can (and should) test it once every month or two. But if you create a landing page for a limited time offer, such actions would be rather pointless. Remember:

• Network traffic in A/B tests is being directed to two website variants
• These variants are being shown alternately – one internet user sees variant A while the other sees variant B at the same time
• The variant with the highest CTR wins!

Why are A/B tests worth it?

While creating a website, you should keep in mind that it’s not a closed process. There is a great difference between creating a website and creating a profitable website. You’ll need A/B test for the latter purpose, but you can use them even if you write your own blog.

What should we test?

The truth is, you can test anything you place on your website. The most crucial thing is that it shouldn’t be more than one element at a time, because it would be nearly impossible to determine what truly changed the result. Almost every website that is designed to bring income should have these elements tested:

• conversion buttons (CTA – call to action)
• headers
• product presentation
• element arrangement
• size and placement of photos
• color palette
• list of benefits and features that should convince a customer to use our product
• style of communication

These are the most common elements that you should do an A/B testing at the beginning of your testing adventure. The list isn’t complete, though. Feel free to test whatever you like. Even the slightest change can make your landing page convert better.

Copy test examples

What’s more efficient when it comes to building headers? You can test some of these: Questions:

• Don’t you want to earn more on your website?
• Do you want to earn more on your website?
• How do you earn more on your website?
• Do you think you can’t earn any more from your website?

Numbers and fractions:

• 19 000 000 Poles have bought this product
• Half of all Poles have bought this product
• 50% of the Polish population have bought this product

Text formatting:

• Landingi.com is the best landing page generator.
• LANDINGI.COM is the best landing page generator.
• Landingi.com is the best landing page generator.

Call to Action:

• Buy access to our service!
• Sign up today!
• Try for free!

We / You / It

• We help create an effective landing page
• You can create an effective landing page
• Landingi.com is an app to create effective landing pages

Emotions and senses:

• Feel the force and make your choice – create a landing page of your own!
• Learn about the benefits of creating your landing page
• It’s wonderful to know that your landing page will make money for you!

Text length:

• Our landing page creator is definitely the cheapest on the market
• The cheapest landing page creator on the market
• Cheapest landing page creator

Punctuation:

• Improve your conversion rate today.
• Improve your conversion rate today!
• Improve your conversion rate – do it today!

Quotes, sources, autoquotes:

• Landing pages improve the effectiveness of an advertisement
• Research shows that landing pages improve the effectiveness of an advertisement
• American scientists claim: “Landing pages improve the effectiveness of an advertisement”.

Does it work?

A while ago, I wrote an article about how to build landing pages that convert. I answered a question there: “Why should I choose a landing page over a regular website?”, and I answered it with a bunch of statistics. We used to have a website with all Landingi features and benefits listed and explained. The conversion rate on this one was around 7.5%. We wanted more of course, so we ran – you guessed it! – an A/B test to see if it’s better to direct visitors to a landing page with all this information. As you can probably imagine by now, it worked nicely. The new page granted us 13.87% conversion. Further changes boosted up the conversion on this landing page to a solid 15% – all this because of A/B Tests. So yeah, it works big time!

How to maintain A/B tests?

Testing the effectiveness of specific landing page elements affects general campaign results undoubtedly. We’ve already mentioned what we should test and how to do it earlier. Read more about it in our article: How to build the perfect landing page form? We’re getting close to the moment in which all plans are set, a landing page is ready to launch and there’s not much left to do, but launch a test. Let’s do it step by step:

1.  First, you should ask yourself a question: What can be the reason for a low conversion rate? Sometimes it’s the color of a CTA button. Sometimes it’s a headline or images. There are plenty of ways to find out what’s causing the vicious cycle of no conversions – e.g. heat maps.

“A heat map is a diagram of a visitor’s engagement on your website. A heat map shows which parts of a landing page are the most interesting for your users.”

2. Prepare two different variants of your landing page. Remember that you’ve got to make them differ with only one element to run a verifiable A/B test.

3. Take your time to choose your target audience. Also, notice that it’s always better to test on a fresh group of customers – they won’t have any past experience with your website, so their behaviour will be clean and reliable.

4. Choose the source of traffic. It may be Facebook, Google search, mailing and so on.

5. Start testing!

A friendly reminder:

• test a maximum of two variants at the same time
• direct the traffic from the same sources
• run tests in the same conditions – it’s the only way they can be reliable

6. Choose the winner! Analyzing the results is not rocket science. You just have to check which combination got better conversion. You’ll know immediately what caused it, because there was only one change (what was it?). Write it down somewhere, it can come in handy later.

7. Run another test! I hope you didn’t think that this is the end! If your first A/B test seemed to be successful that just means there should be more, because with every test your landing page improves! And while it improves, it gets better conversions, better CTR and eventually better sales!

Simply put: A/B testing is not a single action, it’s an ongoing process.

How to run tests in the Landingi platform

Launching A/B tests varies in different platforms. Here’s a step by step how-to for Landingi users. First, go to your landing pages section, click More on the right and select “Run Tests”: Now it’s time to add a variant of your landing page – just click the corresponding button to copy it: Don’t forget to edit the newly created copy – implement the change you want to test. You can also rename it to avoid mistakes. I’ve changed the name of the original to “Test 17 A” and the variant to “Test 17 B”. The next step is setting the test. Now you need to specify when it should end. We recommend running an A/B test for at least 14 days. Of course, you can change an ending condition during the test. Here you can also split the traffic – 50/50 seems like the best solution, but you may freely change it: Now there’s only one thing left: Start the test, and improve your landing page.

A short guide to prepare a test campaign

1. A/B Tests – a baseline study

If you intend to start a test for a whole new campaign, you can skip this stage. But if you’re in the middle of it and you want to improve the result, start the optimization with reviewing a current one. Analyze all sources of traffic and check statistics of your website. Take a closer look at:

• Bounce rate – is it too high?
• Time spent on your website
• Conversion rate – check if it’s below the average in your industry
• The quality of conversions

With this data gathered you can single out the aspects that need improvement. A short amount of time spent on your website could be a sign that your marketing message isn’t correct. Users enter your landing page and they may feel disappointed with the dissonance between what you’ve promised and what they get. Also, it can be a design flaw. No one likes to stay on an ugly website. Finding a solution will get much easier if you identify the problem first. It’s only one of many examples included in InstantShift’s 25 Ugly Websites That Make You Want to Cry.

2. Setting the goal

Before you start A/B testing, you should set the goal. After doing a baseline study you should know what is your biggest problem. Time to point out what exactly  you want to achieve. Choose one goal only:

• more visitors
• more file downloads
• more form fill-outs
• bigger sales

3. Set the hypothesis

The next stage is to form the hypothesis that should then become verified by a test. One goal goes with one hypothesis. Clarify which elements need improvement:

1) Subject of research

Choose the weakest link of your campaign and decide whether you want to improve a landing page, an advertisement or maybe a traffic source instead? You’ll focus on it in a few later stages.

A high bounce rate could be the result of incorrect communication that drives traffic to your landing page. In this case, focus on testing the ad. A low conversion rate, on the other hand, can be a sign of a troubled landing page.

2) Elements of research

Define elements that can affect your message effectiveness:

• ad contents
• header
• CTA button
• offer
• color palette
• form

Choose one of them and change it on your landing page. You’ve got to determine how detailed your test will be. Are you going to replace the whole element or just switch colors?

3) Set the number of variants

Determine how many combinations you’ll compare. If you decide to create a few variants of advertisements of landing pages with different elements, decide if you’re going to use an A/B/C test (3 website versions at the same time) or divide the test into stages. The choice is yours, but notice that if your campaign is rather small, you shouldn’t test too many options simultaneously. Now you know what you’ll test. Express the hypothesis and start testing.

If you’re testing the form, the hypothesis expression can be: “shortening the form will result in a 20% higher conversion rate”.

Let’s take a look at two form variants and try to guess which one guarantees a better conversion rate. Variant A: Variant B: The general rule is: the more form fields, the worse. Especially, when there’s no reason to include some of them (which is exactly the case of variant B). Sometimes you’ll need three or four form fields, and that’s perfectly OK as long as they are necessary, i.e.: for a conference sign-up landing page.

4) Sources of traffic

Determine where your target comes from. Define the group in which you see the potential for converting and think about how you can reach it. You can drive traffic to your landing page using many channels: a search engine, email marketing, display ads, social media and more.

5) Choosing the tool

There are many split testing tools that make testing easier. If you’re testing ads, you can analyze their results in Google Ads in real time. If you’re testing landing pages you’ve got to use a previously created tool. The Landingi platform has got a built-in A/B Testing tool, which I described earlier in the article. You can use it to run tests, see the statistics and choose the winner after a predefined amount of time, visits or conversions.

6) The course of the test

Now it’s the time to set the course of your A/B test. There are some options in our tool that will determine the ending of a test. It could be:

• test length
• number of visits
• number of conversions

The minimum conditions that will let you get trustworthy results is 2 weeks, 1000 visitors or 100 conversions. Of course those numbers depend on industry specification and target audience characteristics. This is why they should be individually chosen for every campaign.

7) Building variants and running the test

If you’ve created some test variants (mentioned in section 3.), you can now run the test. A friendly reminder: Test A and B at the same time and in the same conditions (traffic source). Only this will make them reliable.

8) Reporting and analyzing the results

After some time testing you’ve got to pick the winner. There’s no need to run some advanced results analysis. Just check which combination got the best conversion score and watch out for the goal you’ve set before. Also, do your best to determine what was the direct cause of the winner’s success. You’ll need it in the future. So now you’ve got a winner. That’s the end of one test, but only the beginning of testing. Plan some more research, go back to section 1 and run another one. Create a new hypothesis and study it. Learn more from your successes and failures. Find the optimal combination of elements and don’t be afraid to turn it upside down while running another test. Regular optimization means a more effective campaign.

5 things you should remember while A/B split testing

At the end of this article I want to give you 5 simple pieces of advice you should keep in mind during all future A/B tests. Enjoy! 1) test simultaneously It means you should test two prepared variants of a landing page at the same time. This will minimize the factors that could interfere with results. There are plenty of elements that could negatively affect the reception. For example: the seasonality. Users act differently on work days, during weekends or holidays. The right course of testing should focus on showing two versions of a landing page alternately to user A and user B. If you run tests on different days of week, your results won’t amount to much. 2) a test should last a minimum of 2 weeks The more representative a sample you get, the better. You can’t get it during a three-days test. Take 10 days as the shortest length of testing and 14 days as optimum. 3) get at least 100 conversions / 1000 visits Even if your test is running for 14 days, but there’s still less than 1000 visits, you should prolong it until you get at least 1000 visits or a hundred conversions. 4) test traffic on new clients Your previous clients have developed their habits regarding your website. They like specific parts of it, while avoiding others. They are used to some parts and this is why their judgement will be emotional. You’d do better to test on a new target audience with no previous experience with your website. 5) test constantly After you’re done with one test, start another. Results may be better already, but you want the best. Improve your landing page and therefore your campaign constantly. Run test after test and then throw in another one. Analyze previous statistics, check the bounce rate, conversion rate and time spent on your landing page. Specify exactly what your goal is and make your website better and better! Create your landing pages, optimise and split test them – all in one place!