Multivariate and A/B test FAQs

Testing basics

What is the difference between A/B testing and multivariate testing?

A/B testing

In A/B testing, the marketer is experimenting with a single variable within the campaign (such as email subject lines or message send time). This involves randomly dividing a subset of the audience into two or more groups, presenting each group with a different variation, and observing which variation exhibits the highest conversion rate. Typically, the best-performing variation is subsequently sent to the remainder of the audience.

Multivariate testing

Multivariate testing is an extension of A/B testing, which allows the marketer to test multiple variables at once to determine the most effective combination. For example, you could test the subject line of your email message, the image that accompanies your text, and the color of the CTA button. This type of testing allows you to explore more variables and variation combinations within a single experiment, and obtain insights faster and more comprehensively than A/B testing. However, testing more variables and combinations within a single experiment requires a larger audience to yield statistical significance.

How are A/B test results calculated?

Braze tests all the variants against each other with Pearson’s chi-squared tests, which measure whether one variant statistically outperforms all others at a significance level of p < 0.05, or what we refer to as 95% significance. Across all variants that exceed this significance threshold, the best-performing variant is determined to be the “winner”.

This is a separate test from the confidence score, which only describes the performance of a variant compared to the control with a numeric value between 0 and 100%. Specifically, it represents our confidence that the standardized difference in conversion rate between the variant and control is significantly greater than chance.

Why isn’t the variant distribution even?

The distribution between variants is not always even. Here’s how variant distribution works.

Every time a message is sent in a multivariate campaign, the system independently selects a random option according to the percentages you set and assigns a variant based on the result. It’s like flipping a coin—anomalies are possible. If you’ve ever flipped a coin 100 times, you know that you probably won’t get an exact 50-50 split between heads and tails every time, even though you only have two choices. You might get 52 heads and 48 tails.

If you have multiple variants that you want to split evenly, you also need to make sure the number of variants is a multiple of 100. Otherwise, some variants will have a higher percentage of users distributed to that variant compared to others. For example, if your campaign has 7 variants, there can’t be an even variant distribution since 7 does not equally divide by 100 as a whole number. In this case, you would have 2 variants of 15% and 5 variants of 14%.

Note on in-app messages

When running an A/B test on in-app messages, your analytics may appear to show a higher variant distribution between one variant and another, even if they have an even percentage split. For example, consider the following graph of Unique Recipients for Variant A and Variant C.

Variant A has a consistently higher count of Unique Recipients than Variant C. This isn’t due to variant distribution, but rather how Unique Recipients are calculated for in-app messages. For in-app messages, Unique Recipients are actually Unique Impressions, which is the total number of people who received and viewed the in-app message. This means if a user doesn’t receive the message for whatever reason or decides not to view it, they are not included in the Unique Recipients count, and the variant distribution can appear skewed.

Running and concluding tests

When is the initial test over?

When using Winning Variant for single-send campaigns, the test is over when the Winning Variant Send Time arrives. Braze will deem a variant to be the winner if it shows the highest conversion rate by a statistically significant margin.

For recurring, action-based, and API-triggered campaigns, you can use Intelligent Selection to continuously track each variant’s performance data and continuously optimize campaign traffic toward top-performing variants. With Intelligent Selection, rather than explicitly defining an experiment group where users receive random variants, the Braze algorithm will continuously refine its estimate of the best-performing variant, potentially allowing for faster selection of the top performer.

How does Braze handle users who received a message variant in a recurring campaign or Canvas entry step?

Users are randomly assigned to a particular variant before receiving the campaign for the first time. Each successive time the campaign is received (or the user re-enters a Canvas variant), they will receive the same variant unless the variant percentages are modified. If the variant percentages change, users may be redistributed to other variants. Users stay in these variants until percentages are modified again. Users will only be redistributed for the variants that were edited.

For example, let’s say we have a campaign or Canvas with three variants. If only Variant A and Variant B are changed or updated, then users in Variant C won’t be redistributed because Variant C’s variant percentage wasn’t changed. Control groups remain consistent if the variant percentage is unchanged. Users who previously received messages can’t enter the control group on a later send, nor can any user in the control group ever receive a message.

What about Experiment Paths?

The same applies because the Canvas paths following an experiment are also variants.

Can I take actions to redistribute users in campaigns and Canvases?

The only way to redistribute users in Canvases is to use Randomized Paths in Experiment Paths, which will always randomize path assignments when users re-enter the Canvas. However, this isn’t a standard experiment and could invalidate any experiment results because the control group can become contaminated with treatment users.

Confidence and bias

Does confidence increase over time?

Confidence increases over time if all else holds constant. Holding constant means there aren’t other marketing factors that could influence variants, such as Variant A talking about a 25% off sale that ends midway through the test.

Confidence is a measurement of how confident Braze is that the variant is different from the control. As more messages are sent, the statistical power of the test increases, which would increase the confidence that measured differences in performance are not due to random chance. Generally, a larger sample size increases our confidence in identifying smaller differences in performance between variants and control.

Can control and test group assignments introduce bias to testing?

There is no practical way that a user’s attributes or behaviors before the creation of a particular campaign or Canvas could vary systematically between variants and control.

To assign users to message variants, Canvas variants, or their respective control groups, we start by linking their randomly generated user ID with the randomly generated campaign or Canvas ID. Next, we apply a sha256 hashing algorithm and divide that result by 100, and keep the remainder (also known as a modulus with 100). Finally, we order users into slices that correspond to the percentage assignments for variants (and optional control) chosen in the dashboard.

Why can’t I use rate limiting with a control group?

Braze currently doesn’t support rate limiting with A/B testing that has a control group. This is because rate limiting doesn’t apply to the control group in the same way as the variants, thus introducing bias. Instead, consider using Intelligent Selection, which automatically adjusts the percentage of users that will receive each variant based on analytics and the performance of the campaign.