Multi-arm Bandit Testing with SAS Customer Intelligence 360
From a marketing perspective, websites, mobile apps and emails are designed to maximize one or multiple business goals on behalf of a brand. Examples include:
- How do we get more prospects to click the "Buy Now” button?
- Which subject header attracts the most opens to an email campaign?
- Which version of a creative placed in the hero spot of a website home page obtains the highest engagement rate?
To determine the optimized insight, brands run A/B tests to compare two or more variants of a single interaction experience with customers. A/B tests search for statistical significance by comparing a treatment group(s) and a control group. A key metric is used to validate the difference in performance across the groups, such as average time on site (engagement), purchase rate (e-commerce), or click through rate (advertising).
A balanced A/B test would allocate an equal amount of impressions to each variant, until reaching sufficient sample size. However, traditional A/B test designs cannot adjust how variant impressions are served based on what is observed. Although A/B tests are proven their value for countless brands and use cases, there is one disadvantage. If one of the test treatments is performing better, brands will serve inferior experiences to the control group, in order to obtain statistically significant read. This results in less purchases, email open rates, or weaker levels of engagement.
Figure 1: Optimizing A/B test designs with SAS Customer Intelligence 360
It would be ideal if there was a way to run an A/B test, but not waste opportunities to convert customers with inferior experiences.
Multi-arm Bandit Testing
Imagine going to a restaurant. The intent is to order the tastiest menu item. However, this is challenging because to truly order the best dish, one must order everything available to identify it. The balance of exploitation, or the intent to select an action which has rewarded well historically, and exploration, the desire to choose options which could produce better results, is what multi-armed bandit algorithms were created for.
A multi-armed bandit solution is a variation of A/B testing that leverages machine learning to dynamically allocate customers to variations that are performing positively, while allocating less impressions to variations that are underperforming. The value proposition of multi-armed bandits is the efficient delivery of better results.
Although there are different approaches to addressing the multi-armed bandit problem, one takeaway is for certain. Multi-arm bandit solutions are a self-learning method of approach. Once it is set up in SAS Customer Intelligence 360, the user can let it run hands-off. This feature balances exploration with exploitation. The variant of the test design with the highest known reward (or monetary value) is selected, except when a random action is delivered. A random selection of the overall variants is pulled a fraction of the time. This helps keep the solution fresh as customer behavioral data evolves.
Within the subject of embedded customer analytics, let's walk through a presentation and demonstration together in the video below to address how SAS helps:
- Continuously adapt to customer decisions
- Help automate the customer decision-making process over time
- Optimize decisions
Learn more about how the SAS Platform can be applied for embedded customer analytics here.
