A control group is a segment in a test that continues to receive marketing with no change, so that you can compare results with a segment that is receiving a new campaign, channel outreach, etc.
A control group is a group of users in a test that do not experience the tested variable, and therefore help to create a comparison data set to measure results. If you want to start working with a new channel, or approaching users in a new way, your control group would continue to receive normal marketing, not the new channel or approach. If you see an increase in conversions that’s statistically significant for the tested group, compared to the control group, you can assume that your new approach was effective. However, if there is no significant difference between conversions for the two groups, you may want to try a new approach, because you can’t conclude that the new tactic made a difference.
It’s important that control groups are made up of similar demographics as the rest of your users (so that you don’t accidentally have mostly women in your control group, for example, which adds another variable to the mix that you’re not trying to test). You should also set a hypothesis before you start and focus on a single variable to test, to make the results as easy to understand as possible.
“To test how push notifications enhanced our onboarding campaign, we kept a control group (who received everything but the new push notifications) and that made it much easier to see the great effect the notifications had on the tested group.”