Autotuning Multiple Model Types in Parallel Using Model Studio
Automated machine learning can benefit data scientists of all skill levels, from beginners to seasoned experts. Model Studio offers you automation at different stages in modeling life cycle, however in this post we will delve upon the automated modeling part that includes the process of hyperparameter tuning and algorithm selection.
When you have your data ready for modeling, there are multiple modeling algorithms to choose from, and within each one of them is an abundance of options and hyperparameters to set. Determining the best values of machine learning algorithm hyperparameters for a specific data set can be a difficult and computationally expensive challenge. It can involve a lot of manual trial and error. As algorithms get more complex (neural networks to deep neural networks, decision trees to forests and gradient boosting), the amount of time required to identify these parameters grows. To find the optimal values in an automated manner, automatic hyperparameter tuning (autotuning) can be performed. Machine Learning capabilities in SAS Viya empowers users to automatically tune hyperparameters of modeling algorithms by using a parallel local search optimization framework to ease the challenges and expense of hyperparameter optimization. In addition to this, it also offers an extra-strength version of autotuning in the form of a Model Composer node in Model Studio that enables you to autotune multiple model types in parallel.
The Model Composer Node
The Model Composer node is a Supervised Learning node. It supports both class and interval targets. The Model Composer node performs what is known as combined algorithm selection and hyperparameter tuning (CASH). This feature adds another level of automation to autotuning, which can further enhance your productivity. It enables you to autotune various model types in parallel and performs multiple rounds of autotuning. The number of evaluations that are allocated to each model type in the rounds after the first round (where they are equally allocated) is determined by the accuracy and computational cost of each model type in previous rounds. When the rounds of autotuning are complete, the overall best model (according to your objective function) across model types that has the optimal set of hyperparameters is selected.
Currently, Model Composer node supports autotuning of six models namely Decision tree, Forest, Gradient boosting, Neural network, Bayesian network (class target only), and Support vector machine (binary and interval targets).
Working of Model Composer Node
The Model Composer node determines the best modeling algorithm and its best configuration of hyperparameters based on the training data. It enables you to choose multiple model types to autotune concurrently and perform multiple rounds of autotuning to deliver the top model from each model type. Then among these top models from each model type, the overall best model is selected using the objective function value that depends on the target variable. For a class target, the default is the Kolmogorov-Smirnov statistic, whereas for an interval target, the default value is average square error.
All model types are tuned for a number of evaluations initially equal to nEvaluationsPerRound / nModelTypes. For each subsequent round, a heuristic metric, based on the epsilon greedy or SoftMax policy, is used to re-allocate nEvaluationsPerRound to the nModelTypes (Good model types get more evaluations, bad get fewer.) Suppose there are 4 model types and the total evaluations per round is 100. Each model type gets 25 evaluations (25 different model configurations) in round 1. Then, based on how each model type performs, the allocations are adjusted, which means model type 2 might get 75 evaluations, and model types 1, 3 and 4 might get 8 (total still 100 or less). To begin with, the algorithm tries the number of model configurations (that is, a different combination of hyperparameter values) for each selected model type. In round 1, the Model Composer node adjusts allocations and tunes all the model types again. So the rounds are an outer loop essentially to adjust allocations and tune all the model types. From round 1, the model configuration that produces the best objective function value is passed on to round 2. The algorithm again tries a different combination of hyperparameter values for each model type to improve upon the best configuration objective function value obtained from round 1. If the algorithm finds hyperparameter values that result in improved configurations, then the new set of hyperparameter values is kept. Otherwise, the model configuration from the first round is considered for comparison among all the model types. This happens for all the model types. Finally, the top models from each model type are compared with each other, and the model that results in the best value of the objective function is considered as champion. This can continue further if you choose to have more than two rounds. By default, the node can perform up to five rounds, given the value specified in the Maximum time (minutes) property. Note that this time limit might prevent all specified rounds from finishing.
Automatically Tuning Hyperparameters of Multiple Models
In this demonstration, I will be using Model Composer node to illustrate how optimal values of hyperparameters can be determined by performing autotuning for multiple model types. So, instead of performing autotuning on a single model at a time and repeating it for other model types of your choice, you can automatically tune hyperparameters for multiple model types concurrently and then select the top model for your data.
I am assuming that you are already familiar with the steps required to create a pipeline in Model Studio. If not, check out Build Models with SAS Model Studio | SAS Viya Quick Start Tutorial. I start with a blank template and add a Decision tree node to it. To add a Decision tree node, right click on Data node and select Add child node -> Supervised Learning -> Decision Tree. In the Decision Tree properties pane, turn on the Perform Autotuning property. Also, turn on the Minimum Leaf Size parameter. Enabling this property requests that the optimal values of the available hyperparameters be chosen automatically. Your pipeline should resemble the following:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Select Run pipeline to run all the included nodes in pipeline. Open the results for the Decision tree node. In addition to the usual decision tree model output, the node results include the Autotune Best Configuration table that shows the optimal values for the hyperparameters, the evaluation at which those optimal values were obtained, and the value of the objective function (the Kolmogorov-Smirnov statistic in this example).
Note that when you enable autotuning in Model Studio, the node does not update its properties (the hyperparameter values) after it executes. Therefore, the results do not match the properties shown on the right pane of the node anymore. The best hyperparameters are written to the output—in this case, to the Autotune Best Configuration table.
In addition, you get Autotune Results table, which shows the sets of values of hyperparameters that were tried, the evaluation, the resulting objective function, and the evaluation time.
Finding the optimal hyperparameter values through autotuning is exciting. With this simplified process, would you be interested in exploring more advanced modeling algorithms? Most likely, your answer would be yes, as there is no single model that works well under all circumstances. So, if you decide to experiment with other modeling algorithms, you need to add each modeling node individually and perform autotuning to obtain optimal values of hyperparameters. You can then compare the performance of these models and choose the best one to be consumed in the next step of your analytical life cycle. This might appear to be a repetitive task when you intend to try several modeling algorithms.
To circumvent this repetition, consider using the Model Composer node. It enables you to autotune various model types in parallel and performs multiple rounds of autotuning to select the overall best model for you. Next you learn to use the Model Composer node.
Right-click the Data node and select Add child node -> Data Mining Preprocessing -> Imputation. The Imputation node is used in the process flow to tackle missingness in the data. The default impute value is Count for categorical inputs and Mean for interval inputs. Next, right-click the Imputation node and select Add child node -> Supervised Learning -> Model Composer. In the Model Composer node options panel, keep the settings at their defaults. Note that, by default, the node autotunes the hyperparameters of four model types concurrently using the autotuning options. These four model types are Decision tree, Forest, Gradient boosting, and Neural network. In addition, you can also include Bayesian network (supports class target only) and SVM (supports binary target only) by selecting the corresponding check box.
Click Run Pipeline and examine the results for the Model Composer node.
There are several tables and plots to help you evaluate the performance of each model type and the overall best model. The first table is Top Models, which represents the top models for each model type that is specified and the best value of specified objective function. The overall best model is the one with the best value of the objective function across each model type. In the table depicted below, the gradient boosting model has been chosen as the best model among the top models of each model type based on the objective function value, the Kolmogorov-Smirnov statistic.
The Configuration for Best Model table displays the values of the parameters that were autotuned along with the value of specified objective function for the overall best model.
The Best Configuration for Each Model Type table displays the values of the parameters being autotuned for each model type. The configurations of only the best model from each model type are shown below (partial view):
The Evaluations Performed per Round table displays the number of evaluations that were run per round for each model type (partial view below):
In addition to these tables, the Model Composer node results also include several types of scoring code.
Finally, the Output window shows the SAS output of the model composer run. By default, the node performs five rounds to tune multiple model types concurrently with optimal allocations. Therefore, the output below summarizes the best configuration and tuner summary for each model type in each round.
The Model Composer node run can be very time consuming. In case an analyst wants to apply domain knowledge and experiment with the Autotuning options available in the properties panel, be aware that the node can take significant time to return results. One alternative that reduces this execution time and yet enables you to experiment is to consume the results from the Model Composer node and use them in the individual modeling nodes. In this way, an analyst can even try to improvise on the best configurations suggested by the Model Composer node.
To learn more on automated machine learning, consider taking this course: Advanced Machine Learning Using SAS® Viya®
Find more articles from SAS Global Enablement and Learning here.
