Troubleshooting

Predictive Churn (and any machine learning model) is only as good as the data available to the model. It also is highly dependent on having certain volumes of data to work with.

Potential errors

Not enough data to train

This error message appears when your churn definition is too limiting and returns too few churned users.

To fix this, you will need to change either the number of days and/or actions that define churn to capture more users. Make sure you are using AND/OR filters correctly so as to not create overly restrictive definitions.

Problems with prediction audience size

When building out your prediction audience to fine-tune the kind of use you want your model trained against, you might encounter this message notifying you your prediction audience has too few users:

“Not enough past churners available from the selected prediction audience in the past 7 days to reliably build the prediction.”

If your prediction audience definition is too strict, you might not have a large enough pool of both historic and active users to work with. To fix this, you will need to either change the number of days and type of attributes used in this definition, switch up the actions that define churn, or both.

If your prediction audience continues to be a problem even after switching up your definitions, you may have too few users to support this optional feature. We would suggest trying to build a prediction without the extra layers and filters instead.

Prediction audience size is too big

A prediction audience definition cannot exceed 100 million users. If you see a message saying your audience is too large, then we recommend adding more layers to your audience or changing the window of time it’s based on.

Prediction has poor quality

If your model has a prediction quality of 40% or greater, you are in a great place! But if your prediction quality drops to 39% or less, you may need to edit your churn and prediction audience definitions to be more specific or have different time windows.

If you are unable to meet both the audience size requirement while building your prediction definitions and achieve a prediction quality of greater than 40%, it likely means that the data sent to Braze is not ideal for this use case, that there are not enough users with which to build a model against, or that your product life cycle is longer than our current 60-day lookback window supports.

Data considerations

The following are questions to ask yourself as you set up Predictive Churn. Machine learning models are only as good as the data that trains them, so having good data hygiene practices and understanding what goes into the model will make a big difference.

• What high-value actions lead to retention and loyalty?
• Have you set up custom events that map back to these specific actions? Predictive Churn works with custom events as opposed to custom attributes.
• Are you thinking in windows of time within which you’ll define churn? You can define churn as something that happens in up to 60 days.
• Have you considered times of the year that lead to atypical user behaviors such as holidays? Rapid shifts in consumer behavior will impact your predictions.