How Notum Implemented A/B Testing for Strapi.io

Introduction: A/B Testing at Strapi with Amplitude

Description and step-by-step instructions on how we implemented A/B testing on the Strapi.io website.

For this purpose, we will use Amplitude's Experiment. The Experiment creation goes as follows:

1. Create a hypothesis
2. Pick a metric
3. Create a variant
4. Decide who will see the variant
5. Allocate users
6. Activate your experiment
7. Analyse your results

In the Amplitude dashboard, you can then see the results of the experiments in a very intuitive way:

In this blog post, we will focus on “Create a variant” and its implementation in the front-end app.

Requirements

Create an environment where the marketing team can make A/B testing experiments with minimal to no development required. This needs to be done using the Amplitude Experiment JavaScript SDK.

Solution

You can approach this in multiple ways:

1. You can do an “on-demand” variant, fetching from every component on mount. This approach is more tedious but better for performance when there are many variants.
2. You can do a global wrapper that fetches all the variants on mount. This way, there is a longer initialisation, but after that, the component loads are faster, and you don’t need to handle loading states and other side effects.

The decision was to go with the global wrapper route,

1// _app.js  
2import ExperimentProvider from 'src/components/shared/ExperimentProvider'
3
4...
5<ExperimentProvider>
6...
7</ExperimentProvider>  
8...  

In the components, we introduce the experiment key, with previously defined keys like message or class that will be applied to the components.

If the key is present, the corresponding text or styling is going to be applied to the components.

Example Strapi setup

Above, you see an experiment field as a reference for the Amplitude dashboard.

A/B comparison on the website

The images above show the two versions: A - white button, B - blue button.

Key takeaways

Pros

The solution above is a good solution for a big existing application, where we want to introduce A/B testing.

When a key is present in a component, it will find its variants. If the variant is there, the colour or text, or any other aspect of the elements, can be modified by the marketing team.

Cons

Loading all the variants on page load can be a problem long-term for the page load speed.

If there is a way to prepare the UI to load the variants when the component is actually loaded, that would be a better solution.