Machine Learning with a Rare Target Event Level – Part 5: Tuning Hyperparameters

In this post we will explore the use of automatic hyperparameter tuning to maximize the average precision metric on validation data. This is useful when dealing with imbalanced data, especially with rare target events. This post is the fifth in a series on building machine learning with a rare target; the previous post introduced the average precision metric and compared models built with a few different numbers of synthetic samples. Now we explore tuning the model hyperparameters and the number of added SMOTE samples to improve performance on the average precision metric. Rather than manually tuning the hyperparameters and refitting the model repeatedly we will instead use a derivative free optimization tool to automatically tune them (an autotuning routine).

The solveBlackBox Cloud Analaytic Services (CAS) Action allows any kind of CAS Language (CASL) code to be used to define decision variables and an objective function for derivative free optimization. A link to a course on SAS optimization that discusses black box optimization is included in the references, but for this discussion we will focus on the use of the solveBlackBox CAS Action to autotune machine learning models based on custom user-defined code, defining both the decision variables (hyperparameters we want to tune) and the objective function (the metric we want to improve with autotuning). In this example we will tune the learning rate and number of trees in a gradient boosting model, along with the number of SMOTE samples we will add to the training data. We will do the autotuning to improve the average precision, which we calculate using custom code as well. This illustrates the flexibility of the solveBlackBox CAS Action, allowing us to autotune both data preparation and model training hyperparameters based on a user-defined metric.

We start by making a connection to the CAS Server and the CASUSER CASLIB, and then we load the rare target event data into memory. We also define two macro variables for use later on, a list of input variables (all continuous variables) in the &inputs macro variable and the target variable (‘class’) in the &target macro.

cas;
caslib _all_ assign;

/*load the data into memory for analysis*/
proc import datafile='/path/to/data/creditcardfraud_normalised.csv'
            out=casuser.creditcardfraud
            dbms=dlm
            replace;
    delimiter=',';
run;

%let target='class';
%let inputs={{name='V1'}, {name='V2'}, {name='V3'}, {name='V4'}, {name='V5'},
    {name='V6'}, {name='V7'}, {name='V8'}, {name='V9'}, {name='V10'},
    {name='V11'}, {name='V12'}, {name='V13'}, {name='V14'}, {name='V15'},
    {name='V16'}, {name='V17'}, {name='V18'}, {name='V19'}, {name='V20'},
    {name='V21'}, {name='V22'}, {name='V23'}, {name='V24'}, {name='V25'},
    {name='V26'}, {name='V27'}, {name='V28'}, {name='amount'}};

Next, we prepare the data for modeling by splitting it into training and validation samples, with a partition indicator column identifying the 70% training data (0) and 30% validation data (1) samples. In this example we don’t do any additional data preprocessing (variable selection, feature extraction, etc.) but this would be the block of code for any data preparation that we don’t intend to autotune in this example. We also save the preprocessed data so that we can load it when using the solveBlackBox CAS Action for optimization.

proc cas;
    
    /*split data into training and validation samples, using a partition indicator*/
    action sampling.stratified / 
            table={name='creditcardfraud',groupBy='class'}
            samppct=30
            partind=True
            output={casout={name='creditcardfraud',replace=true},copyVars='ALL'};
run;

    /* save the preprocesed data table */
    action table.save /
            table='creditcardfraud'
            name='creditcardfraud'
            replace=True;
quit;

The first part of our CASL code, the caslInit block, is executed once before the first iteration of the black box optimization routine. In this case we just load the data into memory for use in the subsequent iterations, but we could also do data preparation in this block of code. This code is not run repeatedly by the solveBlackBox CAS Action when trying to improve the user-defined objective function.

/*now for the training code*/
proc cas;

    source caslInit;

        /*load the preprocessed data table*/
        action table.loadTable / 
            path='creditcardfraud.sashdat'
            casout={name='creditcardfraud', replace=True};
        run;

    endsource;

The evalCode is run repeatedly by the solveBlackBox CAS Action as part of the optimization routine, with the goal of tuning the decision variables (selected hyperparameters we will see later) to improve the objective function (in this case the average precision which we will calculate later). The first part of our evaluation code involves adding synthetic samples using the smoteSample CAS Action, the CAS Action version of the SMOTE Procedure we used in previous posts in this series. We use SMOTE to generate a number of synthetic fraud cases (1s) equal to the decision variable nsamples, and then we use the DATA Step to add those synthetic fraud cases to the training data. The optimization routine will repeatedly change the value of nsamples in order to improve model performance (the average precision defined as the objective function).

 source evalCode;

        /*add synthetic samples using SMOTE*/
        action smote.smoteSample / 
            table={name='creditcardfraud',where='_PartInd_ = 0'}
            nominals={'class'}
            classColumn='class'
            classToAugment=1
            seed=919
            numsamples=nsamples
            casout={name='creditcardfraud_smote', replace=True};

        action dataStep.runCode / 
            code="data casuser.creditcardfraud_train;
                      set casuser.creditcardfraud(where=(_PartInd_=0)) casuser.creditcardfraud_smote;
                  run;";

Next we train the gradient boosting model on the augmented data. In this example we use the default settings for the gradient boosting model, but include the hyperparameters ntree, and learningRate as decision variables, ntrees, and lr to be autotuned by the optimization solver. We store the output model in an in-memory table for use in scoring.

/*train the gradient boosting model*/
        action decisionTree.gbtreeTrain / 
            table={name='creditcardfraud_train'}
            inputs=&inputs
            target='class'
            ntree=ntrees
            learningRate=lr
            casOut={name='GB_Model', replace=True};

We score the validation data using the trained model and then run the assess CAS Action to calculate ROC and confusion matrix information for the scored data. We are interested in the confusion matrix (TP, FP, TN, and FN) information across a range of cutoffs so that we can calculate precision and recall for use in the average precision calculation. The cutstep option is set to 0.001 so that we end up with 1,000 values for precision and recall across a range of cutoffs from 0 to 1.

/*score the validation data using the trained model*/
        action decisionTree.gbtreeScore result=score / 
            table={name='creditcardfraud',where= '_PartInd_ = 1'}
            modelTable='GB_Model'
            casOut={name='Valid_Scored', replace=True}
            copyVars=&target
            encodename=True
            assessonerow=True;

        /*assess the validation data to calculate confusion matrix across a range of cutoff values*/
        action percentile.assess / 
            table='Valid_Scored'
            inputs='P_class'
            response='class'
            event='1'
            cutstep=.001
            casOut={name='valid_assess', replace=True};

Next, we use the SAS Time Series modeling procedure (TSMODEL) to calculate the average precision by summing across the 1,000 cutoffs calculated earlier. We must use the TSMODEL Procedure because the distributed nature of SAS Viya means that calculations across subsequent rows of data may not be consistent unless we treat them as a sorted time series. We are not actually analyzing time series data, so if we were working in SAS 9 we could use the lag() function in a DATA Step to calculate average precision, but we use SAS Viya to take advantage of the solveBlackBox CAS Action.

We start by using the DATA Step to add an observation number variable (obs) to the dataset based on the cutoff value to use as a time ID in the TSMODEL Procedure. Then we read in the confusion matrix values as time series into the TSMODEL Procedure, sorting them by the cutoff value. We use the time series modeling language to calculate the average precision by summing across the series adding up precision values multiplied by differences in recall values. Recall that the average precision is defined as follows:

Average Precision = ∑_n(Recall_n - Recall_n-1) Precision_n

The TSMODEL code calculates precision and recall at each cutoff value and then calculates a rolling sum in the ap_sum variable. At the end of the series this value will contain the full average precision for the scored validation data, and we save it as a scalar in the table ap_scalar.

/*calculate the average precision using the SAS time series functionality*/
        action dataStep.runCode / 
            code="data casuser.valid_assess_roc;
                    set casuser.valid_assess_roc;
                    obs = round(_cutoff_,0.01)*100+1;
                    run;";

        action timedata.runtimecode / 
            table='valid_assess_roc'
            series={{name='_TP_'},{name='_FP_'},{name='_FN_'},{name='_TN_'},{name="_cutoff_"}}
            arrayOut={table={name='apcalc', replace=True},arrays={{name='precision'},{name='recall'},{name='AP_calc'}}}
            scalarOut={table={name='ap_scalar', replace=True}, scalars={{name='ap_sum'}}}
            interval="obs1"
            timeID={name='obs'}
            code="do i = 1 to _LENGTH_;
                        precision[i] = _TP_[i] / (_TP_[i] + _FP_[i]);
                        recall[i] = _TP_[i] / (_TP_[i] + _FN_[i]);
                        AP_calc[i] = (recall[i-1]-recall[i])*precision[i-1];
                        ap_sum += AP_calc[i];
                  end;";

We use the average precision value that we just calculated as the objective function for the black box optimization, so we extract it from the ap_scalar table and provide the numeric value only (stripped of any formatting or structure) as a response to the optimization routine. This involves first fetching the ap_scalar table and storing it in a CASL variable before extracting the actual numeric value for average precision (ap_sum), saving it as a double precision number and then sending a dictionary to the optimization solver indicating that this number is the objective value. This then ends the CASL source code that the optimization solver will run repeatedly, trying to improve the objective value by changing the decision variables.

/*we want to evaluate the model using the Average Precision (AP) on the validation data*/
        action table.fetch result=scalar / table='ap_scalar';

        ap = scalar['Fetch'][1]['ap_sum'];
        ap = (double)ap;
        f['objVal']=ap;
        send_response(f);    
        
    endsource;

Now that we have the prepared data and the code we want to iteratively run we call the solveBlackBox CAS Action to perform the derivative-free optimization. At this point we could do our own experiments by running the code in a loop with different values for the hyperparameters, but even with our limited number of 3 hyperparameters we would need a triply nested for loop, causing an enormous amount of computation. The derivative free black box optimization solver will perform a more efficient search than we could do manually, but it still takes time to run and cannot provide optimality guarantees like traditional optimization solvers. In this case we tune the three decision variables (the hyperparameters are the number of synthetic samples to add, the number of trees in the gradient boosting model, and the learning rate for the gradient boosting model), providing upper and lower bounds for the variables to limit the search space. We maximize the average precision value, and print out the final solution.

/* now we try to optimize the hyperparameters of the model, in this case the number of trees in the gradient boosting model */
    action optimization.solveBlackBox / 
            decVars={{name='ntrees', type='I', lb=100, ub=1000},
                     {name='lr', type='C', lb=0.001, ub=1},
                     {name='nsamples', type='I', lb=50, ub=3000}}
            obj={{name='objVal',type='MAX'}}
            func={init=caslInit, eval=evalCode}
            primalOut={name='solution', replace=True};
quit;

proc print data=casuser.solution;
run;

arzitin_resultsbb.png

arziti_bbsol.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

The best solution found had 100 trees, a learning rate of 0.034662, and includes 579 synthetically generated fraud cases. Although we are performing optimization, we can’t call this an optimal solution since we cannot prove optimality, and it’s likely we could find a better solution with a longer search time. This solution has an average precision of 0.83427, and involved 10 iterations of the black box solver, which is 251 total evaluations of the code we provided (we fit 251 different gradient boosting models and calculated 251 average precision values, selecting the hyperparameters that yielded the highest average precision.

There are many different ways to improve model performance when working with rare target event data, so going directly to autotuning all of the relevant hyperparameters with a black box optimization routine is not the most efficient use of computation, but it can be useful after trying many different ways to improve model performance after identifying a good range of values for the hyperparameters. One goal of this post is to illustrate the use of the solveBlackBox CAS Action to automatically tune unusual hyperparameters in models, like the number of synthetic samples to add to the data, and to showcase custom metric calculations for autotuning. As hyperparameters and evaluation metrics become more popular they are added to the built-in autotuning tools for machine learning models, but it is convenient to have the flexibility provided by the solveBlackBox CAS Action to automatically tune hyperparameters for uncommon hyperparameters or metrics.

Previous blogs in this series:

References:

Find more articles from SAS Global Enablement and Learning here.

Machine Learning with a Rare Target Event Level – Part 5: Tuning Hyperparameters

Ready to see what SAS Viya Copilot can do?

SAS AI and Machine Learning Courses