Beginner's guide to setting up your first A/B test with Optimizely

Beginner's guide to setting up your first A/B test with Optimizely

In this tutorial, I’ll show you how to setup and run your first A/B test in Optimizely. We’ll start with a very simple experiment to test to see whether our call-to-action (CTA) is more effective for conversions as a button instead of a link on your page. For further ideas for A/B testing, see the links at the end of this post.

Experiments listing

On the home page, you’ll have a list of all your previous experiments (if any). You can filter these by status or by typing in the name of the experiment. Ensure you give your experiment a relevant name, so it is easy and relevant to find later!

Experiments listing

New Experiment

Give the experiment a name (make it something memorable and relevant to what you are testing) Enter the URL of the page where the experiment will sit.

Give the experiment a name

Naming variables

Rename variables to describe what the test is doing so it makes sense when you come back later on. Click on the dropdown and select Rename Variation to do this.

Rename Variation

Rename Variation

For our test, we wanted see if the “Join us” call-to-action would be more effective for conversions as a button instead of a link within the bullet points as show below. This page has since been re-designed but is a nice reminder that A/B testing is key to constantly improving and iterating your pages based on customer feedback.

Our first A/B test

And here is the final button we implemented. You’ll notice we also changed the wording from “Join us” to “Join CAF Give As You Earn” as it was much more descriptive and assured the user what they were signing up for.

Final button we implemented

To place your button on the page, it is best practice to click on a link, button or wherever you’d like to add your new element, and click Select Container and then choose the div or container that is most relevant. In this case I had a plain div around my button.

Selecting the Division “div” makes an outline around the button like so:

Here it is again once selected:

We do this because selecting the whole div and editing the HTML is much easier for editing and positioning things on the page.

I can then edit the styles around the div to position the button easily. Don’t use the Move & Resize option as it’s unreliable - edit the HTML instead.

Edit Element… → Edit HTML

Give your element a class name

This is so important as it will allow us to identify our element so we can create goals against it. Make it “a unique class name” and this should be done before you set up a goal.

Right-click → Edit Element → Edit Classes

Creating our first goal

Select the button → Track Clicks → Create New Click Goal

Give the goal a name (make it relevant to what you are tracking!) and in this case we are tracking clicks on that button.

Next click on Advanced at the bottom of the page

In the selector box we should see our newly created class name.

Original tab

Back in the original tab, in our example the two things you want to test against each other should have the same class, in our example, join-us-link.

Navigate to another page within Optimizely

You can switch to another page within the Optimizely editor. Go to Options –> Editor –> Settings and type in the URL to navigate to this page.

On our Join us page there is a download link which downloads a PDF of an application pack. We’d like to track how many people click this so let’s create a goal for this.

Give your element a class name - make that “a unique class name”

Click on the link → Edit Element → Edit Classes

Complete the following fields like this:

Start the experiment

Click on Start experiment to put your test live!

Now the fun part starts, you need to make sure the 2 versions of our test are working correctly so open up a new window (preferably in another browser or Optimizely will log you out). You’ll need to clear your cookies and open up the page where your test sits. Repeat the process of clearing cookies and refreshing until you see both versions - it may take a number of attempts :)

In Internet Explorer, to clear your cookies go to Settings → Internet Options → General tab → under browsing history click Delete and make sure cookies are ticked.

You’ve started your first A/B test - congrats! Now keep an eye on the results over the next few days to make sure everything is running as expected.

Further reading

The Results page for Optimizely Classic

71 A/B Testing Ideas

How to Run an A/B Test to Improve Your CRO

How the Obama Digital team optimized to increase donation conversions 49% over 20 months

How Netflix does A/B Testing

Neil Berry

Senior UX Designer @cafonline. Previously Senior Designer @newsfromfta. I blog about UX, design, and the latest tech.

Do you want to understand the best way to convert visitors to customers in 2018?

Let's design something incredible together.