Feature Extraction in SAS Model Studio, Part 2: Robust Principal Components Analysis

[#1]

This post is part 2 in a four-part series on feature extraction techniques in SAS Model Studio, where I’m exploring practical methods for preparing data for predictive modeling. In Part 1, I introduced Principal Components Analysis (PCA) as a way to reduce dimensionality and handle correlated inputs.

In this post, I’ll build on that foundation by introducing Robust Principal Components Analysis (Robust PCA). While PCA is effective in many situations, it has one key limitation: it can be sensitive to outliers. Robust PCA is designed to address that limitation, making it a valuable tool when working with real-world data.

Why Go Beyond PCA?

As discussed in Part 1, PCA works by identifying patterns of variation in the data and summarizing them into a smaller set of components. However, PCA assumes that the data is relatively “clean”. If you’ve ever worked with real data, you know that’s rarely the case. Real-world datasets often contain outliers, data entry errors, or even rare but extreme observations. Such observations can distort the components generated by PCA because the method tries to account for all variation among the inputs, including unusual or misleading values. This is where Robust PCA becomes useful.

A Simple View of Robust PCA

At a high level, Robust PCA takes the idea of PCA one step further by separating the data into two parts; mathematically these two parts can be expressed as matrices. I’ll discuss a bit more on the matrix stuff later. The first part of the data is the structured component. It captures the underlying patterns shared across most observations. You can think of this as the “normal” part of the data. The second part of the data is the sparse component. This is the portion of the data that has the unusual or extreme values; the potential outliers. You can think of this as the “problematic” part of the data.

As a comparison, PCA attempts to explain everything with a single structure. Robust PCA attempts to separate what’s typical from what’s unusual. This separation allows Robust PCA to build principal components based on the true underlying structure while isolating outliers instead of letting them influence the results of the components.

Why Use Robust PCA for Feature Extraction?

Robust PCA can be especially valuable when preparing data for modeling because it addresses challenges that standard PCA does not. Robust PCA is outlier resistant. Extreme values are separated rather than influencing the main components. Robust PCA provides cleaner feature representation. The resulting components better reflect consistent patterns in the data. Robust PCA results in improved model stability. Models trained on robust components are less sensitive to unusual observations. Finally, Robust PCA provides a built-in insight into anomalies. The sparse portion of the decomposition can help identify potential outliers. In short, Robust PCA not only reduces dimensionality, but it also improves the quality of the transformation.

Without getting too deep into the math details, (not the purpose of this post!) Robust PCA relies on matrix algebra computations to provide the solution. The original data matrix is decomposed into a sum of two matrices: a low rank matrix and a sparse matrix. Using matrix notation, this is typically denoted as the following:

M = L0 + S0,

where M is the input data, L0 is the low rank matrix used for feature extraction, and S0 is used for anomaly detection. The matrix decomposition in Robust PCA is essentially an optimization problem subject to the constraint above. Principal components are then computed on the matrix L0 after the outliers have been removed. Just as with PCA, Robust PCA is an unsupervised technique, so the target variable is not being used in the matrix operations.

Example: Applying Robust PCA in SAS Model Studio

Let’s look at how Robust PCA can be implemented in SAS Model Studio using the Feature Extraction node. As in Part 1 of this series, we start with a dataset from a telecommunications company that is trying to predict customer churn. The data includes many interval inputs, making it a good candidate for feature extraction. Recall that the Feature Extraction node uses only interval inputs.

Here’s the starting pipeline:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

The Feature Extraction node ignores observations with missing values, so the imputation node is used to make all observations complete. Let’s take a look at the properties panel for the Feature Extraction node. First, at the top, I’ve changed Feature extraction method to Robust PCA.

Once Robust PCA is selected, several properties control how the decomposition is performed. There are two main parts of the properties panel for Robust PCA. The first is found under RPCA Options:

One important property is the Penalty weight, which is related to a component of the optimization process which is denoted mathematically by λ (Lambda). Lambda is a penalty applied to the sparse matrix, S0, during the optimization solution. The Penalty weight property determines how the method balances the complexity of the low-rank matrix with the sparsity of the outlier component. A higher penalty weight leads to a sparser matrix S0, which translates to fewer outliers. Smaller values of the penalty weight decrease the rank of the low rank matrix L0. In many cases, the default setting works well, as it is based on the size of the data. The default value of the penalty term lambda is one over the square root of the size of the training data (λ = 1/sqrt(n)).

The second portion of the properties panel for Robust PCA is found under the Additional RPCA and SVD Options:

The Maximum rank property is similar to the Maximum number property discussed in my prior post on PCA. Depending on the number of input variables in the data, the Maximum rank property indicates the maximum number of Robust PCs selected by the node. I’ll keep all options at their default for now.

After running the node, the results provide insight into both parts of the decomposition. Before we dig into that, let’s explore other parts of the results. Here’s the Eigenvalue Plot (a.k.a. the Scree plot) that shows the final number of Robust PCs selected:

The vertical line on the right side of the plot indicates that 53 components were selected. However, there are possible “elbows” in the plot we might want to investigate. Let’s consider the 2-component solution (circled on the left-hand side of the plot).

We can explore the cumulative amount of variation explained by using the drop-down menu to change the focus of the plot to Cumulative Proportional Eigenvalue.

We see that the 2-component solution accounts for nearly 92% of the variation in the input space. This is very exciting that such a small number of components accounts for such a large amount of variation in the data!! (Yes, I am a nerd! But if you have read any of my prior posts, you already know this.)

Now, let’s go to the Output window to explore the low-rank versus anomalous underlying structures of the data.

First, note that the size of the training data (the Number of Observations Used) is 39,590. This is how we get to a Lambda value of 0.0050258. Recall that λ = 1/sqrt(n). Under the Results Summary we see the rank of the low-rank matrix is 53 (hence, the 53 components selected above) and that the sparsity of the sparse matrix is 0.1513. This sparsity value means that only about 15% of the values in the sparse matrix are 0, indicating about 85% of the data are outliers. Although we expect outliers in the data, this value does not seem realistic.

Let’s make some adjustments to the properties to see if we can get a more reasonable solution.

Here I’ve changed the Penalty weight to 4. Realize that the Penalty weight is not lambda itself, but a value multiplied by lambda. (This is essentially a safeguard that prevents users from putting in ridiculous values of lambda.) Recall that although the default value of lambda is often optimal, that does not mean the default is best for all situations. This change should increase the sparsity of the sparse matrix, which should translate to fewer outliers.

Here’s the Cumulative Proportional Eigenvalue plot after changing the penalty weight property:

Rather than considering the 2-component solution mentioned above, a 7-component solution looks promising as it accounts for over 87% of the variation of the input data. Let’s check the Output window.

Given our updated penalty weight of 4, the new value of lambda is 0.0201 (not shown above) which is calculated by 4 x 1/sqrt(n). The rank of the low-rank matrix is now 68 and the sparsity of the sparse matrix is 0.9874. This now means that about 1.2% of the data are classified as outliers; a much more realistic solution compared to that based on using the default settings.

Let’s go back to the properties for a final time and make a change so that we arrive at the 7-component solution.

Under Additional RPCA and SVD Options, I changed Maximum rank to 7 so that only the top 7 Robust PCs are passed to supervised nodes and I’ve changed Component prefix to RPC so the newly generated features are clearly labeled as being Robust PCs.

Running the node again and looking at the Scree plot reveals that 7 components will be passed on as new features.

I connected a Gradient Boosting node to the Feature Extraction node and built the model. We see that several of the newly generated features, denoted with RPC prefixes, are selected among the top inputs by the model, as shown in the Variable Importance table.

Summary and Looking Ahead:

It is well established that during data preprocessing, feature extraction can be a critical step when building effective predictive models. Robust PCA extends PCA by separating structure from outliers in input data. It is especially useful for real-world data that contains noise or extreme values. It produces both a low-rank representation of the data and a sparse component useful for identifying anomalies. The resulting components are derived from the low-rank portion of the data which has outliers removed. In SAS Model Studio, Robust PCA is easily implemented using the Feature Extraction node.

I have only two more parts to do in this four-part series on feature extraction: Singular Value Decomposition and Autoencoders. Stay tuned for the next post in the series soon!

For more on Robust PCA:

Training:

Advanced Machine Learning using SAS Viya

Communities Posts:

Your Toolbox for Unsupervised Machine Learning in SAS Viya

Discover SAS Visual Data Mining and Machine Learning Procedures

4 ways to classify feature engineering in SAS Viya

Find more articles from SAS Global Enablement and Learning here.