SAS 360 Direct: Holdout Control Groups
A holdout control group is a common marketing test. SAS Customer Intelligence software has included features to enable this type of test analysis for many years. SAS 360 Direct has built-in features to easily separate out a fraction of the targeted population of a Direct Marketing task to go into a holdout control group.
The purpose of this post is to introduce the concept of what is a holdout control group, and how the software can be used to set up this type of test. Like most marketing tests, the purpose is to measure the effectiveness of campaigns.
Holdout Control Groups – Testing Campaign Effectiveness
Holdout control groups are typically a very small subset of recipients from your segment who do not receive the marketing offer. For example, for a specific segment that is receiving a 50% off coupon, a portion of the segment that is the holdout control group will not receive the offer. Some organizations have a default or standard size for the holdout control group, such as 5% of the segment population. As we will discuss later, the software can be enabled with a default size in the form of a percentage or count of target subjects within a segment.
The purpose of the test is to compare the response of the holdout control group with the response of the population that received the offer. This is one method to test the results and effectiveness of a campaign. The example below illustrates the conversion rates between the offered and holdout groups. One can perceive from this example that the offer was effective because it resulted in a much higher conversion rate for the offered group. But, results will be relative to your market, business, and offerings so below is an example to help illustrate the purpose of a holdout control group.
If there is not much gap in the difference between the conversion rates, that could possibly prove the campaign was not very effective and possibly not worth the budget spent. But again, the results will be analyzed and conclusions drawn relative to the goal of a campaign.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Configuring Holdout Control Groups in SAS 360 Direct and Direct Marketing Tasks
Default settings for holdout control groups can be set for a business context using General Settings. However, that is optional to apply and they are considered “default” which means they can be changed to other options within the task itself. In the illustration below, you can see there is a Testing tab within the Direct Marketing task. The first step is to enable holdout control groups for the task. This could be enabled by default – thanks to the Business Context settings.
Then you can set the test parameters for each segment that is used by the task. Different holdout sizes can be applied to each segment if desired. For the One Household segment below, the size of the holdout group was set to 5% of the segment population. That size could come from the from either the default applied in the business content settings or from the default applied in the individual task settings. The task settings can be accessed by clicking the Settings button also highlighted below.
Note that the default size – if there is one – can be simply changed within the task to another value. As mentioned earlier, the size can be set to a specific count or a percentage. The size option (percentage or count) can also be changed (if there was a default applied). As mentioned, the default size can also be applied at the task level – but again, that value can be changed here within the tab for each segment. Here is an illustration of the task Settings, which is basically a mirror image of the business context settings (note also the feature below to Reset to default business context settings).
What happens if the segment size is too small to populate the holdout control group? The Settings above show there are 2 choices of action to choose from should this happen: select the holdout control group not be populated and created, or the actual task can be set to fail execution and no exports are created.
The next setting below that has to do with stratified sampling. Stratified sampling can be used to ensure the holdout control group is representative of the total segment population based on specified characteristics. For example, gender could be selected from available criteria as a characteristic to use for stratified sampling. This would cause the system to enable the distribution of gender to be the same for the holdout control group as it is for the segment population as a whole. The three options if the sample size is too small for stratification are: not include a holdout control group, fail execution of the task, or use random sampling and ignore stratification selections.
Export Settings
In the export settings (within the Export Content task tab) there is an option to include holdout control group members in the export table settings:
Control Group Flag and Control Group Type are fields that should added to the export:
If the Include holdout control group members option box is selected then members of the holdout control group will be included in the export and be flagged by the selected fields. This will be useful if analysis will be conducted using other applications.
Results of the test can be visualized in the Insights tab. However, as with all Direct Marketing tasks, the campaign results events will need to be captured and loaded into the system for this to be visualized in the Insights tab:
For more details and information on using holdout control groups in Direct Marketing tasks, and for SAS 360 Direct itself, please visit these sites below:
Courses:
- SAS 360 Direct Marketing Tasks
- Building Segments using SAS Customer Intelligence 360
- SAS 360 Direct – Defining the Marketing Data and Business Contexts
- Custom Tasks and External System Tasks with SAS Customer Intelligence 360
Find more articles from SAS Global Enablement and Learning here.
