Feature Extraction in SAS Model Studio, Part 3: Autoencoders
This post is part 3 in a four-part series on feature extraction techniques in SAS Model Studio, where I’m exploring practical methods for preparing data for predictive modeling. In Part 1, I introduced Principal Components Analysis, or PCA, as a way to reduce dimensionality and handle correlated inputs. In Part 2, I discussed Robust PCA, which extends PCA by separating the underlying structure in the data from unusual or extreme values.
In this post, I’ll introduce autoencoders, a feature extraction method based on neural networks. While PCA and Robust PCA construct features using linear combinations of the original inputs, autoencoders can capture more complex, nonlinear relationships in the data. I’ll discuss the basic idea behind autoencoders and then provide an example of implementing one with the Feature Extraction node in SAS Model Studio.
Why Go Beyond Linear Feature Extraction?
PCA and Robust PCA are both useful methods for reducing the dimensionality of input data. However, the principal components created by these methods are based on linear combinations of the original inputs. This works well when the important relationships in the data are primarily linear.
Of course, relationships in real-world data are not always linear. In fact, one might argue that in real-world data relationships are usually not linear. The effect of one input might depend on the value of another input, or a pattern might change across different ranges of a variable. When these types of nonlinear relationships are present, a linear feature extraction method might not capture all the useful patterns in the data.
Autoencoders address this limitation by using a neural network to learn a compressed representation of the input data. This allows them to capture patterns that PCA and other linear methods might miss.
Does that mean autoencoders are always better than PCA? Of course not! The appropriate method depends on the data, the modeling objective, and the amount of complexity that is justified. However, autoencoders provide another useful option when the input data contains complex relationships.
A Simple View of Autoencoders
An autoencoder is a neural network that is trained to reproduce its inputs. It is an unsupervised neural network, meaning, no true target variable is used. Just the inputs. That might sound like an unusual objective for a neural network. If the inputs are already known, why build a neural network to predict them?
The real purpose is not simply to copy the inputs. The purpose is to force the network to learn a useful, compressed representation of the data while attempting to reconstruct the original values.
An autoencoder has two primary parts: an encoder and a decoder. The encoder takes the original inputs and compresses them into a smaller number of hidden features. The decoder takes those hidden features and attempts to reconstruct the original inputs.
Between the encoder and decoder is a middle-hidden layer that is sometimes referred to as the bottleneck layer. The bottleneck typically contains fewer units than the number of original input variables. Because the network must reconstruct the original inputs using only the information that passes through this smaller layer, it is essentially forced to retain the most useful patterns in the data.
The hidden values created in the bottleneck layer become the new features that can be passed to a predictive model. The image below illustrates the bottleneck (i.e., the middle-hidden layer) between the encoder and decoder parts of the model.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Here is one way to think about the process. Suppose the original data contains 70 inputs and the bottleneck layer contains 10 units. The encoder compresses the information in those 70 inputs into 10 new features. The decoder then tries to use those 10 features to reconstruct the original 70 input values. If the reconstruction is reasonably accurate, the 10 hidden features must contain much of the important information from the original inputs. The autoencoder is not designed to perfectly reproduce the inputs. In fact, there is some “noise” added into the training process to prevent the encoder from perfectly reproducing the inputs. One method to do so is known as dropout which is briefly discussed below.
Unlike the principal components generated by PCA, these new features are not restricted to being linear combinations of the original variables. The activation functions and hidden layers of the neural network allow an autoencoder to learn nonlinear patterns.
Available Structures in Model Studio
An autoencoder model should always have an hourglass shape. In Model Studio, there are only two options for the structure in terms of the total number of hidden layers. The options are 3 hidden layers or 5 hidden layers, and the choice is controlled in the Number of hidden layers property:
The default number of hidden layers is 3. When 3 hidden layers are used the default number of neurons in the first and third layers are 100 each and this is controlled by the First and last hidden layer: number of neurons property. When 5 hidden layers are used an additional property appears: Second and fourth hidden layer: number of neurons. The default is 50. These default values maintain the hourglass shape for autoencoder models whether they have 3 or 5 hidden layers.
Another important property that affects the structure of the model (and ultimately how many new features are passed on) pertains to the number of neurons in the middle layer. This was referred to above as the bottleneck layer. The number of neurons in this layer is controlled in the Middle hidden layer: number of neurons (features) property.
The default value is 10. Since the number of neurons in the middle-hidden layer provides the number of features that are passed on from the node, the million-dollar question is: How many neurons should be used in the middle layer? The answer is problem specific. Trial and error could be used on a selection of various values, and those results (features based on different numbers of bottleneck neurons) could be passed on to supervised models. Of the different models built based on different numbers of inputs, see which model performs best on holdout data. Another option is autotuning. However, at the current time, autotuning for autoencoders is only available through code, not directly in the Model Studio interface. This topic deserves its own discussion, which I’ll write about at some future time, when the current series on feature extraction is complete.
Why Use Autoencoders for Feature Extraction?
Autoencoders offer several potential benefits during data preprocessing.
First, as stated above, they provide nonlinear feature extraction. This is arguably the primary advantage of an autoencoder over PCA and other methods. An autoencoder can learn more complex relationships among the original inputs instead of being limited to linear combinations.
Second, autoencoders perform dimensionality reduction. When the bottleneck layer contains fewer units than the number of original inputs, the network produces a compressed representation of the data. Models can then be trained using these new features instead of the full set of raw inputs.
Third, autoencoders can provide a degree of noise reduction. The network attempts to learn the general structure in the input data rather than reproduce every small fluctuation.
Finally, autoencoders can help produce more efficient predictive models. Replacing a large number of inputs with a smaller collection of extracted features can reduce the dimensionality of the modeling problem and potentially improve model generalization.
One drawback is lack of interpretability. Just as supervised neural networks are typically treated as black box models which are not interpretable, the same is true of features generated from autoencoders. If the final model needs to be interpreted, autoencoders may not be good candidates for generated features.
Preventing an Autoencoder from Overfitting
Because an autoencoder is a neural network, it is subject to many of the same overfitting concerns as other neural network models.
If an autoencoder is too complex, it might learn how to reproduce the training observations extremely well without learning a compressed representation that generalizes to new data. In the most extreme case, the network could learn something close to an identity function, essentially memorizing the inputs instead of discovering useful patterns.
Several methods can help prevent this.
Weight decay penalizes large network weights during training. L1 regularization can encourage sparse weights, while L2 regularization shrinks weights toward zero. Both approaches can limit the complexity, and thus prevent overfitting, of the network.
Early stopping monitors performance on validation data and stops training when the validation error no longer improves. This prevents the network from continuing to adapt to patterns that are specific to the training data.
Dropout randomly excludes a proportion of inputs or hidden units during each training iteration. This prevents the network from depending too heavily on any one input or hidden unit and can improve its ability to generalize.
It is also important to create an appropriately sized bottleneck. If the bottleneck contains too many units, the network might not achieve meaningful dimensionality reduction. If it contains too few, the network might discard important information and produce poor reconstructions.
Example: Constructing Autoencoder Features in Model Studio
Let’s look at how an autoencoder can be implemented in SAS Model Studio using the Feature Extraction node.
As in the previous posts in this series, I’ll use data from a fictitious telecommunications company that is trying to predict customer churn. The data contains more than 120 variables. And unlike PCA and RPCA which are applied to interval inputs only, autoencoders are based on all inputs, both interval and categorical.
Here is the starting pipeline:
After the Data node, an Imputation node is added to handle missing values. The Feature Extraction node ignores observations with missing input values, so the Imputation node helps ensure that complete observations are available for training the autoencoder.
I then added a Feature Extraction node after the Imputation node. Both the Imputation and Feature Extraction nodes are available in the Data Mining Preprocessing group.
Let’s look at the first part of the properties panel for the Feature Extraction node.
For the Feature extraction method property, I selected Autoencoder. I also left Reject original input variables selected. With this property selected, the original inputs (interval and categorical) will be assigned a role of rejected, and the newly created hidden features are passed to successor nodes as inputs.
Once Autoencoder is selected as the feature extraction method, additional properties become available for controlling the neural network.
By default, the autoencoder uses three hidden layers, as shown in the Number of hidden layers property above. The default architecture contains 100 neurons in the first hidden layer, 10 neurons in the middle layer, and 100 neurons in the third hidden layer. These values are seen above in the Middle hidden layer: number of neurons (features) property and the First and last hidden layer: number of neurons property.
The middle layer is the bottleneck. Therefore, keep in mind, that if the middle layer contains 10 neurons (the default), the Feature Extraction node produces 10 new features to be used by subsequent models.
The default activation function is the hyperbolic tangent, or tanh, function which is also commonly used for supervised neural networks.
The properties panel also contains options related to regularization and training. These include L1 and L2 regularization, the maximum number of iterations, and the stagnation value used for early stopping. Note the property Create Validation at the bottom of the properties pane. This property is turned on by default. When turned on, this property means that a temporary validation data set is created from the existing training data. This temporary validation data set is used during the optimization process for the autoencoder and then is combined back into the original training data after the node is run.
Additional optimization properties are shown after expanding SGD Optimization Options:
I’ll keep all properties at their default settings.
After running the Feature Extraction node, several results are available for evaluating the autoencoder. One important result is the Iteration Plot which shows, by default, Validation Error across training iterations.
The iteration plot can also show the change in the Objective function or Loss by changing the pull-down menu next to View chart. We see a noticeable decrease in the Validation error starting around iteration 20. Also, as the Maximum iterations property is set at 300 and it appears that less than 40 iterations were run, the training did stop early based on the validation data. To see what caused early stopping, we can look at the Output window.
One of the tables in the Output window is for Model Information.
First, note that the NNET procedure is being used by the node to create the autoencoders. We also see that the value of the Mean Squared Error for Validation for the final model is 18.68434 (rounded). To see the iteration where this occurred, we can look at the Iteration History table.
We see the final model came from iteration 33, based on the minimum Mean Squared Error for Validation value, which is provided in the third column. Training stopped early because of the optimization property Stagnation for early stopping which has a default value of 5. Early stopping yields the model at iteration 33 as the final model because there were five successive iterations where validation error did not improve starting from that point.
Now let’s explore the new features produced by the node. The Output data Variables table shows that 10 new interval columns with a role of input are available (Hidden1-Hidden10) and that all other variables, categorical and interval, are now rejected.
And below is a sample of what the actual values of some of the new features look like for a few of the observations.
Each observation receives a value for every new hidden feature. These values are calculated from the original interval and categorical variables as information passes through the encoder portion of the network.
To see the new features used in a supervised model, I connected the Feature Extraction node to a Gradient Boosting node and ran the model.
Based on the Variable Importance table, The Gradient Boosting model selected only 2 of the 10 inputs generated by the autoencoder.
Summary and Looking Ahead
It is well established that feature extraction can be a critical data preprocessing step when building effective predictive models. Autoencoders extend feature extraction beyond linear methods by using a neural network to learn compressed representations of the original input data.
The encoder compresses the inputs, the middle bottleneck layer contains the newly extracted features, and the decoder attempts to reconstruct the original inputs. Because an autoencoder can learn nonlinear relationships, it can capture patterns that PCA and Robust PCA might miss.
Well, we’re getting close to the end of this journey on Feature Extraction methods in Model Studio. I now have only one final part remaining in this four-part series: Singular Value Decomposition, also simply known as SVD. Stay tuned for the final post in the series coming soon!
For More on Autoencoders
Training
Advanced Machine Learning Using SAS Viya
Deep Learning Using SAS Software
Additional Resources
Autoencoder analysis using PROC NNET and neuralNet action set
Find more articles from SAS Global Enablement and Learning here.
Ready to see what SAS Viya Copilot can do?
Visit the Tips & Tricks page for setup guidance, demos, and practical examples that show how Copilot supports your workflows.
