A/B test Advanced Personalization

A/B testing, also known as split testing or bucket testing, compares two versions of a web page, app, or other product to decide which one performs better.

For the Advanced Personalization feature, A/B testing involves comparing two versions of search results. These search results are based on the effect of the personalization re-ranking level you set.

By conducting an A/B test, you can find the ideal level of personalization that suits your business and lets you maximize user engagement and satisfaction.

This feature isn’t available on every plan. Refer to your pricing plan to see if it’s included.

Prepare to A/B test Advanced Personalization

Before setting up an A/B test, make sure you have properly configured Advanced Personalization on the dashboard:

Alternatively, you can use the Advanced Personalization API to configure Advanced Personalization.

Launching an A/B test

You can launch an A/B test directly from the Advanced Personalization dashboard. The process involves the following steps:

1. Select the index on which you want to A/B test personalization. This is the index for which search results will be compared. You only have access to indices that are configured to use Advanced Personalization.

2. Define the personalization re-ranking level for the A/B test. This is the level of personalization you want to A/B test against the personalization re-ranking from the configuration.

3. Set the percentage of traffic. This is the share of searches that will be boosted according to the personalization re-ranking level set.

4. Set the duration of the test. This is the maximum duration of time the A/B test will be live after you start it. For the best results, we recommend that you set a duration that is more likely to lead to high confidence in results.

5. Launch A/B test. This triggers the launch of the A/B test. Your users will now receive personalized search results based on your setup.

Viewing the results of an A/B test

You can view the results of all Advanced Personalization A/B tests in the A/B testing dashboard.

A/B test real-time personalization

Real-time personalization is a beta feature according to Algolia’s Terms of Service (“Beta Services”).

You can also use A/B testing to determine if real-time personalization is the right option for your application. After testing historical personalization, you can test the impact of real-time personalization to see if this approach leads to more engagement.

You can measure the impact of real-time personalization in two ways.

1. If you aren’t yet using historical personalization, you can measure the combined impact of real-time personalization and historical personalization.
2. If you are already using historical personalization, you can either:
   • Measure the incremental impact of real-time personalization compared to historical personalization alone
   • Stop using historical personalization for all users by setting personalization re-ranking to none and measure the combined impact of real-time personalization and no personalization.

Measure combined impact of real-time and historical personalization

1. Ensure that you have setup real-time personalization in the Advanced Personalization dashboard.
2. Follow the real-time personalization integration guide.
3. Repeat the steps in the launching an A/B test section.

The results of the A/B test will show you how real-time personalization combined with historical personalization affects user engagement compared to no personalization.

Measure incremental impact of real-time personalization

During the beta, A/B testing the incremental impact of real-time personalization compared to historical personalization alone isn’t natively supported. However, you can still measure this by setting up an A/B test using a tool of your choice.

The results of the A/B test will show you how real-time personalization affects user engagement compared to advanced personalization.

A/B testing tools

You can implement an A/B test for real-time personalization using a tool of your choice. Some popular options include:

• AB Tasty
• LaunchDarkly
• Optimizely
• Split.io
• VWO

Implement A/B testing

Every A/B testing tool has its own setup process, but the general steps are as follows:

1. Create a new A/B test. Set up a new A/B test in your chosen tool.
2. Define the test groups. Split your users into two groups.
   • Control group: users who won’t experience real-time personalization.
   • Test group: users who will experience real-time personalization.
3. Implement the test. Use the A/B testing tool to control which group each user belongs to. If the user is in the test group, compute the real-time personalization profile in your application. If the user is in the control group, do not compute the profile.
4. Track user interactions. Monitor user interactions in both groups to gather data on engagement metrics.
5. Analyze results. After running the test for a sufficient period, analyze the data to determine if real-time personalization led to improved engagement compared to the control group.

Considerations

• To avoid skewing the results, make sure to exclude returning users from the A/B test.
• Ensure that the A/B test runs long enough to gather sufficient data for analysis.
• Consider factors such as sample size, confidence intervals, and statistical significance when interpreting the results.

During beta, real-time personalization will activate only for new users, while historical personalization will apply for returning users. This is to ensure that metrics for real-time personalization can be collected and analyzed.

For full implementation details, refer to the documentation of your A/B testing tool and the real-time personalization integration guide.

Next steps