This reference article goes over how to measure user retention for users who have performed a selected retention event in a specific Canvas. By knowing how your users are being retained after sending a message, you can measure the effectiveness of your Canvases.
User retention is one of the most important metrics for any marketer. Keeping engaged users coming back for more indicates that business is healthy. Braze allows you to measure user retention right on the Canvas Analytics page.
Please note that Retention Reports are available for both Canvases and campaigns, currently excluding SMS and API-triggered campaigns.
Run a Retention Report
Get started by visiting any Canvas in your Braze Dashboard, and select a date range for your report. Selecting an appropriate date range is crucial because of the way it affects the retention reports. This report will include all users who initially entered the Canvas during this window, and of those users, the data of those that performed their retention event during the date range will appear in the report. To select a date range, you must navigate to the upper right-hand corner of the Canvas Analytics page, here, you may select various ranges or set a custom range for your report.
Next, select the Analyze Variants button at the bottom of the page. From here, you can analyze your variants, check out your funnel report, and view your retention report. Canvas Retention shows you the rate at which any user who has received this specific Canvas has performed a Retention Event (specified by you on the Retention Report) over the 30 days from the time they received the Canvas.
Once you’ve selected a Retention event, click “Run Report” to start the query.
This query may take a few minutes to run, depending on how much data needs to be retrieved to generate the results. If it takes too long, you’ll see a notification asking you to Retry loading the report again (just by clicking the “Retry” button in the body of the report.) You may need to wait up to five minutes before the report will load.
Once the report is generated, it can’t be re-run with the same Retention event for 24 hours. You will always see a timestamp of when the report was last generated, and an option to Regenerate, if it’s been more than a day. You can, however, change the Retention Event and re-run the report to look at the impact of the Canvas on different KPIs.
The report will only show days down the left column on which the Canvas was sending messages. For some Canvases, that may mean the report only shows one day if it was a one-and-done Canvas. If it’s recurring or triggered, you may/will see multiple days down the left.
Our Retention Report offers both a Rolling Retention and Range Retention formula. To view your Canvas report with one of these retention types, select the Rolling Retention or Range Retention radio dial listed under the Type of Retention.
Rolling retention measures how many users come back and do the retention event on or after any of the days listed across the top of the report. So, if a user started a session between day 3 and 7, the user will be counted as retained under the “3 days”, “1 day”, and “0 days” columns. Any user who is counted as retained after the 30-day mark from when the Canvas was sent on the date in the left column will be counted under the “30 days” column in that row.
Note that a user who completes the event multiple times during a window of 30+ days will be counted as part of multiple time frames. For instance, a user who completes a session after 1 day will be incremented in the columns for >0 and >1; if they then complete the event after 3 days, they will again be incremented in the prior columns (>0 and >1), which could result in the retention rate exceeding 100%.
Range retention measures how many users come back in the range of days listed across the top of the report. So, if a user started a session between days 3 and 7 and then again on day 13, they would be counted as retained under both “Day 3-7” and “Day 7-14” ranges.
Braze Retention Reports Components:
- Users Column: The value shown is the number of unique users that performed the start action within the selected time frame; the count of users for the present day will be excluded since it is being calculated.
- Cohort Z Rows: Shows the days in which the Canvas was sending messages.
- Day X Columns: Days spanning between 0 and 30 days at various increments.
- All Users Row: Also known as the Report Summary Row, summarizes the retention data for the entire time period. Note that if a user has entered the Canvas in multiple cohorts, their results will be counted twice here.
- Percentages/Numbers: Shows the percentage/number of users who performed the event X or more days after receiving the Canvas on Z day. These percentages are the weighted average percentages. Incomplete values will be denoted by an asterisk.
- Date Range: Set on the Canvas Details page, the date range includes all users who received the Canvas during this window, and of those users, the data of those that performed their retention event during the date range will appear in the report.
- Units: You can adjust the units between the percentage of users and the number of users in the upper right-hand corner of the chart, specific units may prove to be more significant when judging the impact of a Canvas.
- Color Mapping: In your retention report, higher percentages/number of users are assigned darker shades of blue. Lower percentage/number of users, lighter shades of blue. This is done to help users visualize this data.
- Retention Report Graph: This graph summarizes the results for all cohorts for the selected date range.
How to Read a Range Retention Report
Range Reports are some of the most intuitive reports to read. They clearly state, of all the users in a cohort, what percentage of those users performed the retention event within a given date range. For example, in the image shown below, referencing the All Users Cohort, on date range “Day 0 (0-24hrs)”, 35.71% of the cohort performed the retention report. If a user performs multiple retention events within multiple date ranges, they will be counted as retained for each range.
How to Read a Rolling Retention Report
The way to read the retention report chart for a day 3 column would be Y% or Y number of users (based on units chosen) performed the event 3 or more days after receiving the Canvas on day Z.
As another example, referring to the table above, on 25th of March, a total of 38 users performed the retention event. Day 0 retention was 68.42%, meaning that 68.42% of users performed the retention event 0 or more days (on Day 0 or later) after entering the Canvas. Day 7 retention was 57.89%, meaning 57.89% of users performed the event 7 or more days (on Day 7 or later) after entering the Canvas.
This information can be useful if you want to know the percentage of users who have and have not used your product 30+ days after first use. A percentage/number value in the day 30 column tells you the percentage of users who returned on day 30 or after.
Note that a user who completes the event multiple times during a window of 30+ days will be counted as part of multiple time frames, which could result in retention rates of greater than 100%. For instance, a user who completes a session after 1 day will be incremented in the columns for >0 and >1; if they then complete the event after 3 days, they will again be incremented in the prior columns (>0 and >1). This could lead to the retention rate exceeding 100%, particularly for the earlier time periods.
Performance by Variant
Viewing your Retention Report by variant allows you to compare rolling retention for each variant or message variation for the selected time period, as well as the Control Group. This report can be viewed by toggling the Retention Report from “Entire Canvas” to “By Variant”.
Report by Variant Use Cases:
- Have some variants or experiments in which the results seem like a wash or have no statistical significance? Take another look and see if one or the other had a longer-tail impact.
- See what retention looks like if you didn’t send that message by digging into the Control group’s retention data.
Braze Retention Reports By Variant Components:
- Date Range: Set on the Canvas Details page, the date range includes all users who received the campaign during this window, and of those users, the data of those that performed their retention event during the date range will appear in the report. Each day the retention rate, percentage change from the control group, and confidence are measured.
- Retention Rate: Shows the retention rate by variant. The retention rate is equivalent to the number of users that performed the retention event divided by the total users that have received the canvas.
- Percentage Change from Control: Quantifies the percentage change per variant from the control group.
- Confidence: Braze compares each variant’s conversion rate against the control’s conversion rate with a statistical procedure called a Z Test to calculate a confidence percentage. This percentage signifies how confidently that variant is performing better than the control group.
- Units: You can adjust the units between the percentage of users and the number of users in the upper right-hand corner of the chart, specific units may prove to be more significant when judging the impact of a canvas.
- Variant Graph: This graph summarizes the results by variant for the selected date range.
Things to Look For in your Retention Reports
Retention Reports are easy to generate, yet challenging to interpret and act on. To help aid marketers, we have assembled a couple of topics/questions to consider when looking at your Retention Reports.
- Day-of-week trends for recurring Canvases (e.g Do Monday cohorts perform better than Saturday cohorts?)
- Where does the impact start to decline? This could be a signal that a new Canvas that targets users at that point in time is needed as another boost to retention.
- Am I seeing Canvas fatigue?
- Did that optimization to this Canvas I made X days ago have a positive impact?