The automated feature engineering template is the newest addition to the collection of templates in SAS Model Studio on SAS Visual Data Mining and Machine Learning 8.3. A gentle introduction to feature engineering and why automaton is desired can be found at my recent blog post "Automate your feature engineering".   By using automated feature transformation and extraction techniques, this new template automatically creates newly engineered features. The idea is to automatically learn a set of features (from potentially noisy, raw data) that can be useful in supervised learning tasks without the need to handcraft new features.   This template creates new features in three steps: Performing level encoding for high-cardinality variables Creating new features by using the “Best” transformation, PCA / SVD, and autoencoder methods Comparing the predictive performance of the newly engineered features A more detailed explanation of the three steps is as follows:   Step 1 – Performing level encoding for high-cardinality variables       The first node in this step is the SAS Code High Cardinality. The SAS Code node is my favorite in Model Studio because it provides infinite opportunities for experienced programmers to incorporate their own code into their pipeline for more customized tasks while still enjoying the user-friendly, point-and-click environment of Model Studio for other tasks. You can easily see the content of this node by clicking on it and then opening the Code Editor. The following is contained in this SAS Code node:     This SAS Code node first identifies the high-cardinality variables (nominal variables that have too many unique levels) as nominal input variables that have between 20 and 1,000 levels. You can easily modify this range by updating the values for minlevels and maxlevels. Then, it specifies a numeric transformation (TRANSFORM=LEVELENCODE), level encoding only for those variables. Note that the SAS code specifies the level encoding transformation in the metadata. To apply this transformation, you need to run a Transformations node (a data pre-processing node). Therefore, the connected Transformations node’s role is simply to implement the level encoding transformation that is already specified in the metadata. This is why level encoding is not specified again in the Transformation node.   Level encoding is a simple transformation that converts nominal features to numeric features. It is especially useful for dealing with high-cardinality variables because these variables are often taxing for most machine learning algorithms in terms of computing resources. Level encoding is the simplest nominal-to-numeric transformation; it first orders the levels of a nominal variable in alphabetical order, and then assigns numbers (starting from 1) to each level in ascending order. More efficient nominal-to-numeric transformations, including target-based transformations (e.g., weight of evidence [WOE]) and other similar transformations, are available as part of the “Data Preprocess Action Set” in SAS® Viya®.   Step 2 – Creating new features by using the “Best” transformation, PCA / SVD, and autoencoder methods     Step 2 consists of the following nodes, which apply three different automated feature engineering techniques:   Transformations –  Best: This node uses the Transformations node to specify the “Best” transformation for all interval variables. For each interval variable, this method compares single variable transformations (such as inverse transformation, standardization, centering, and log transformation) based on a ranking criterion (such as correlation with the target) and selects the transformation that has the highest ranking. For a more detailed explanation of this method, see “Best transformation – a new feature in Model Studio 8.3”   Feature Extraction – PCA: This node uses the Feature Extraction preprocessing node to specify the Automatic feature extraction technique for the interval input variables. If the total number of interval input variables is less than or equal to 500, automatic feature extraction is equivalent to principal component analysis; otherwise, it is equivalent to singular value decomposition (SVD).   Feature Extraction – Autoencoder: This node specifies the autoencoder feature extraction technique for creating new features. This technique uses all the input variables (interval and nominal) for feature extraction. An autoencoder is an unsupervised learning technique whose objective is to learn a set of features that can be used to reconstruct the input data. Briefly, a neural network is trained by setting the target neurons equal to the input neurons. There are multiple layers and a bottleneck in the middle layer, so the network is forced to learn a reduced-dimensional internal representation of the inputs before reconstructing them in the output layer. In this template, the middle-hidden layer is set to 10, which means that ten new features will be created.   Step 3 –Comparing the predictive performance of the newly engineered features   To compare the performance of the five different feature sets (three automatically engineered sets, the original set with level encoding for the high-cardinality variables, and the original set without level encoding), the five feature sets are used as inputs for the gradient boosting algorithm. We chose to use the gradient boosting algorithm because it is a very effective supervised learning algorithm that often outperforms other algorithms in terms of predictive accuracy. Automatic hyperparameter tuning (autotuning) is turned on to find the optimal hyperparameter settings of the gradient boosting algorithm so the comparison between feature sets is more fair and not dependent on the hyperparameters. However, keep in mind that autotuning comes with an additional computing cost. If this step takes too long to run, you can change the autotuning settings, or simply turn it off and use the default hyperparameter settings.   The Model Comparison node, at the bottom of the preceding figure, compares the five gradient boosting models and reports performance based on various metrics for existing partitions of the data (e.g. training validation and test sets).   Usage   To use this template more effectively, you first need to increase the maximum class level in Model Studio (the default value is 20) to include high-cardinality variables in the analysis. You can increase this level only when you create a New Project by clicking the Advanced settings. The next figure shows that the Maximum class level is increased to 1,000.       It is important to remember that using this template does not guarantee that one of the newly created feature sets will perform better than the original features for your data, because every data set is unique and there is no guarantee that these techniques will work. Instead, the goal of this template is to show an example that you can follow to create different automatically engineered feature sets by using many other nodes and tools provided in Model Studio and testing their performance in a similar way with minimal effort. Automation through these templates allows you see some simple ideas first to check if there is any value in your datasets before you invest more time, and it can help your company to make quick decisions with lower costs. ... View more

My previous tip on cross validation shows how to compare three trained models (regression, random forest, and gradient boosting) based on their 5-fold cross validation training errors in SAS Enterprise Miner. This tip is the second installment about using cross validation in SAS Enterprise Miner and builds on the diagram that is used in the first tip.   In addition to comparing models based on their 5-fold cross validation training errors, this tip also shows how to obtain a 5-fold cross validation testing error; so it provides a more complete SAS Enterprise Miner flow (shown below).   First a quick note about how k-fold cross validation training and testing errors are calculated: k-fold cross validation training error is calculated by using the predictions on the whole training set which is obtained by combining the k sets of cross validation holdout predictions. k-fold cross validation testing error is calculated by using the average of k different sets of test predictions that come from the k trained models of cross validation. Following is a step-by-step explanation of the preceding Enterprise Miner flow. You can run this process flow by using the attached xml file. The Data Source node HMEQ_TRAIN includes about two-thirds of the HMEQ data, in which the Role property is specified as Train. The HMEQ_TEST node includes the remaining one-third of the data, in which the Role is specified as Test. In both nodes, the target variable is BAD, whose level is binary. Similar to the first tip, to make sure that looping through the Start Groups and End Groups nodes perform k-fold cross validation on the training and test set properly, you need to do some simple tricks. For the training set, this involves using the Transform Variables node as explained in step 3. You need to create 5 replications of test set. In the first test set, the _fold_ variable takes a value 1, in the second test set it takes a value 2 and so on. The following statements in SAS Code node (Test Data X 5) create these five test sets. Note also that it first defines a unique ID for each row in the original test set. The IDs will be used in the next SAS Code nodes (those that come after the End Groups nodes) for averaging the five test set predictions per ID. data temptest; set &EM_import_TEST; ID = _N_; run;   data &EM_EXPORT_TEST; set temptest (in=in1) temptest (in=in2) temptest (in=in3) temptest (in=in4) temptest(in=in5); if in1 then _fold_= 1; else if in2 then _fold_=2; else if in3 then _fold_=3; else if in4 then _fold_=4; else if in5 then _fold_=5; run;  The Transform Variables node (which is connected to the training set) creates a k-fold cross validation indicator as a new input variable, _fold_ which randomly divides the training set into k folds, and saves this new indicator as a segment variable. More information about this node can be found in the first tip. The Control Point node establishes a control point within the process flow diagram. The next three nodes a Start Groups node, a Modeling Node, and an End Groups node train models for each of the five training sets of 5-fold cross validation (each set omits one fold) and obtain predictions on the holdout (omitted) sets. They also obtain predictions on the test set for each of the five trained models of 5-fold cross validation. For each modeling algorithm, the SAS Code 2 node averages the test scores that come from the 5 trained models of 5-fold cross validation by executing the following statements: proc sort data=&EM_IMPORT_TEST out=&EM_EXPORT_TEST; by ID; run; data &EM_EXPORT_TRAIN; set &EM_IMPORT_DATA; run; data test1 test2 test3 test4 test5; set &EM_IMPORT_TEST; if _fold_ = 1 then output test1; if _fold_ = 2 then output test2; if _fold_ = 3 then output test3; if _fold_ = 4 then output test4; if _fold_ = 5 then output test5; run; data &EM_EXPORT_TEST; merge test1(rename=(P_BAD1 = P_BAD_1)) test2(rename=(P_BAD1 =P_BAD_2)) test3(rename=(P_BAD1 = P_BAD_3)) test4(rename=(P_BAD1 = P_BAD_4)) test5(rename=(P_BAD1 = P_BAD_5)); by ID; P_BAD1 = (P_BAD_1 + P_BAD_2 + P_BAD_3 + P_BAD_4+P_BAD_5)/5; run; The Metadata node assigns a new role of “Prediction” for the average predictions of the test set, “P_BAD1”. This new role will be used for performance metric calculation by the Model Comparison node. The Metadata node also rejects the other test set prediction variables with the “Predition” role, P_BAD_1, …, P_BAD_5, so that the Model Comparison node knows which one exactly to use for calculating test set errors. The last node, Model Comparison, compares the gradient boosting, regression, and decision tree models based on the 5-fold training and test set cross validation errors. It provides the following output table, in which the training and testing errors are actually the 5-fold cross validation training and the 5-fold cross validation testing error. Note that you can obtain the cross validated predictions of the test set by saving the exported data (&EM_EXPORT_TEST) of the SAS Code 2 node.   If you run this flow diagram or replicate this analysis for your own data, make sure that you run each Start Groups/End Groups block separately, because multiple looping actions do not work at the same time.   Thanks a lot to Ralph Abbey for his help in putting this together. ... View more

Cross validation is a widely-used model validation technique to estimate how accurately a predictive model will generalize to an independent data set. There are two main uses of cross validation: hyperparameter tuning and model assessment. This post will briefly discuss the use of cross validation in hyperparameter tuning before focusing on using cross validation for model assessment and showing how to compare models based on their cross validation errors using the Start Groups and End Groups nodes in SAS Enterprise Miner.   In hyperparameter tuning, cross validation is used to select the suitable flexibility of a model during model building. For example, when building a neural network model cross validation can be used to find optimal hyperparameter values (e.g., number of hidden layers, number of neurons, learning rate, momentum of stochastic gradient decent algorithm, etc.)  Hyperparameter tuning based on cross validation can be done automatically using the new Autotune statement available in a number of SAS® Visual Data Mining and Machine Learning procedures (PROCs FOREST, GRADBOOST, NNET, and TREESPLIT).   In model assessment, cross validation is used to compare different models that have already been built using the full training data. Suppose you built several models using various algorithms and hyperparameter settings and now you want to compare these models by estimating their prediction power. The basic idea in calculating cross validation error is to divide up training data into k-folds (e.g. k=5 or k=10).  Each fold will then be held out one at a time, the model will be trained on the remaining data, and that model will then be used to predict the target for the holdout observations.  When you finish fitting and scoring for all k versions of the training and validation data sets, you will obtain holdout predictions for all of the observations in your original training data. The average squared error between these predictions and the true observed response is the cross validation error.   In SAS Enterprise Miner, Start/End Groups nodes were originally implemented to stratify an analysis based on a stratification variable. However, with a couple of simple tricks these nodes can be used along with the Model Import node to obtain cross validation error of a model.  You can even compare several models based on their cross validation errors using the Model Comparison node.   Suppose you fit a model on your full training data using the Gradient Boosting node in SAS Enterprise Miner for the following set of the hyperparameters (Niterations=50, Shrinkage=0.2, Train proportion=60, etc):   Now you can calculate cross validation error of this model by running the following flow:   1. Use Transform Variables node to create a k-fold cross validation indicator as a new input variable (_fold_) that randomly divides your data set into k-folds. Make sure to save this new variable as a segment variable. For example, for 5-fold cross validation, Formulas of the Transform Variables node should look like this:       2. In the Start Groups node, specify the “Mode” as “Cross-validation” and in the Gradient Boosting node make sure to use the same parameter settings that you used in your original boosted trees model. Run until the End Groups node.   While the Start/End Group nodes manage to create k versions of training data and calculate fit statistics of the training data, they do not actually calculate the cross validation error from scoring the holdout observations using these fitted models. However, if you check the score code generated by the End Groups node, you can see that it generates the correct score code to calculate the cross validation error. You can view this score code by first clicking the Results of the End Groups node, then on the top menu click View>>SAS Results>>Flow Code. However this readily available score code can be used by another node (such as Model Import node or SAS Code node) to obtain the cross validation error.   3. Attach the Model Import node and run the whole path. The Train: Average Squared Error column in the Results of the Model Import node is the k-fold cross validation error of your original boosted trees model that you trained by using the full training data.   If you are comparing multiple models based on their cross validation errors, your flow (attached as a zip file) should look like this:     Following table shows part of the output table that is produced by the Model Comparison:       Note that because the Model Import node is used, cross validation error is listed as Train: Average Squared Error.  But do not let the ‘Train:’ part confuse you --the Model Import uses the score code generated by the Start/End Groups node in the way we specified in (2), so it is actually the cross validation error. The output table above shows that cross validation error of the gradient boosting model is the smallest. If you choose this model to make prediction for a new data set, make sure to use the score code generated by your initial modeling node which builds the model on the full training set, instead of the k models that are built by the Start and End Groups nodes for the purpose of calculating the cross validation error.   I build on this diagram in another tip, Assessing Models by using k-fold Cross Validation in SAS Enterprise Miner, which shows how to obtain a 5-fold cross validation testing error, providing a more complete SAS Enterprise Miner flow.   ... View more