Multivariate Testing with SAS Customer Intelligence 360
There's an endless set of ideas (or hypotheses) that can be tested in experiential marketing, and every brand has a strategy for optimizing customer interactions. A/B testing is the default for most organizations, as it is the more commonly used technique. But there is a time and a place for multivariate testing (MVT) as well. Let's begin by addressing the difference between A/B and multivariate testing.
A/B testing is a method of marketing optimization in which the conversion rates of two (or more) elements — variant A vs. B — are compared to one another using targeted audiences in channels like web, mobile app, email, or direct. Targeted users are allocated to one flavor of the test design, and by recording the way customers (or prospects) interact — the content they view, the buttons they click, or the purchases they make — brands can determine which variant is most effective.
The business case for A/B testing is frequently summarized as:
- Brands can test more dramatic design changes
- Take less time to run than multivariate tests due to smaller sample size requirements
- The results are easier to explain
It is reasonable to hear practitioners state A/B testing is the least complex method of experiment design. It is a versatile methodology, and when paired with a commitment to iterative learning cycles, it will help brands make improvements to customer experiences. However, it is important to remember that it will not reveal any insight about the interaction between presented elements of a customer interaction. For readers comfortable in performing exploratory data analysis, this is the difference between bivariate and multivariate analysis. The implication on potential insights is vast.
Figure 1: Multivariate testing and experiential marketing
In contrast, multivariate tests are about measuring interaction effects between independent elements to see which combination works best. By using the same core mechanism as A/B testing, while comparing a higher number of variants, it reveals more insight about interaction effects. As in an A/B test, targeted audiences are split between different versions of an experience. The purpose of a multivariate test is to measure the effectiveness each unique recipe has on a business goal.
Sounds romantic, right? However, the biggest obstacle of multivariate testing is the amount of customers needed to complete the test. Most technology vendors will claim these test designs need to be full factorial, and attempting to learn about numerous elements at once can quickly add up to a large number of possible combinations.
Here are a few multivariate testing methods:
Although full factorials provide the most benefit from a data collection perspective, sample size volume and impractical waiting periods are the concern. As for fractional factorials, the trade off is smaller sample size requirements and acceleration to test insights, but measurement precision decreases.
Over a number of years SAS has developed a solution to this problem. This is contained within the OPTEX procedure, and allows testing of designs for which:
- Not all combinations of the factor levels are feasible
- The region of experimentation is irregularly shaped
- Resource limitations restrict the number of experiments that can be performed
- There is a nonstandard linear or a nonlinear model
The OPTEX procedure can generate an efficient experimental design for any of these situations and stereotypical challenges of multivariate tests can be resolved because it applies:
- Constraints on the number of variants that are practical to test
- Constraints on required or excluded combinations of content
The OPTEX procedure is highly flexible and has many input parameters and options. This means that it can cover different customer experience scenarios, and it’s use can be tuned as circumstances demand. From an embedded analytics perspective, SAS Customer Intelligence 360 provides the analytic heavy lifting behind the scenes, and the software users only need to make choices for business relevant parameters.
Sound too good to be true? It was always a problem that involved analytical sophistication, and although every vendor claims to do AI, there is a magnitude of difference in the authenticity of that statement.
Let's walk through a presentation and demonstration together in the video below to address how SAS uses multivariate testing to help brands:
- Continuously adapt to customer decisions
- Help automate the customer decision-making process over time
- Optimize decisions
Learn more about how the SAS Platform can be applied for embedded customer analytics here.
