Machine Learning with a Rare Target Event Level – Part 4: Improving Model Performance

[#1]

This post is the fourth in a series on building machine learning models with a rare target. The previous post focused on evaluating model performance using the Precision-Recall curve and the area under that curve (AUPRC). This post will focus on experiments to improve the AUPRC for rare target event models and will also compare the AUPRC metric to a similar metric, the Average Precision (AP). We discuss consideration for calculating the AUPRC and demonstrate the calculation of AP using SAS Viya tools.

In the previous post we compared a baseline model (using the original imbalanced data) with event-based sampling and SMOTE approaches to balance the data. In both cases we aimed for fully balanced data, so 50% fraud cases and 50% non-fraud cases.

• For event-based sampling this means we ended up with a dataset with 688 observations (344 real fraud cases and 344 randomly sampled real non-fraud cases).
  • This dataset might be a bit too small; we threw away a lot of good non-fraud cases.
• For SMOTE this means we ended up with a dataset with 398040 observations (344 real fraud cases plus 198676 synthetically generated fraud cases and 199020 real non-fraud cases).
  • This dataset includes all the real non-fraud cases, but it might be too heavily weighted with synthetic data since only 0.17% of the fraud cases are real data as opposed to synthetic data.

As we saw in the previous post, the baseline model with the original data distribution outperformed both the event-based sampling and the SMOTE approaches in terms of the AUPRC. These approaches are sometimes helpful when trying to improve model performance but don’t always work well when we just try to balance the data without thinking about the consequences. In this case event-based sampling to a 50/50 sample eliminated too much useful information contained in the non-fraud cases, and creating a 50/50 sample with SMOTE overwhelmed the real fraud cases with too many synthetic cases. There is nothing particularly important or special about a 50/50 sample, so we can experiment with different sample balances to try to improve AUPRC (noting that AUPRC is a cutoff independent measure, so we don’t have to worry about the percentage of fraud and non-fraud cases in the dataset when comparing performance). We will focus on SMOTE to see if we can improve the AUPRC by choosing a better number of synthetic samples to generate. It’s challenging to know in advance the best number of samples to synthetically generate so we will experiment with an increasing number of SMOTE samples and evaluate AUPRC for each number.

The following code continues from the code in the previous posts, so if you plan to copy it and run it be sure to run the code in the previous 3 posts. The full code file can be made available by request.

We start by defining a SAS macro that takes as an input the number of SMOTE samples to generate and add to the training data and plots the precision-recall curve with the AUPRC as an output (it also displays the frequency of fraud and non-fraud cases in the training data). Note that this macro assumes you have run all the code from the previous posts in the series to load and partition the training/validation data.

/*Start by creating a SAS macro to traing and evaluate a model with an imbalanced sample but a user-defined number of additional synthetic samples*/
%macro compare_auprc_smote(num_smote);
    proc smote data=casuser.train seed=919;
        input &inputs / level=interval;
        input &target / level=nominal;
        output out=casuser.smote_samples;
        sample numSamples=&num_smote augmentvar=&target augmentlevel="1";
    run;

    /*we have to merge back the new samples with the original training data*/
    data casuser.train_smote;
        set casuser.train casuser.smote_samples;
    run;

    proc freq data=casuser.train_smote;
        table class;
    run;

    /*now we train a gradient boosting model on the SMOTE modified sample*/
    proc gradboost data=casuser.train_smote outmodel=casuser.smote_model seed=919 noprint;
        input &inputs / level=interval;
        target &target / level=nominal;
    run;

    /*score the validation data using the trained baselines model*/
    proc gradboost data=casuser.creditcardfraud inmodel=casuser.smote_model noprint;
        output out=casuser.smote_scored copyvars=(class _partind_);
    run;

    /*adjust model predicted probabilities based on the smote sampling we performed, this correction will scale the predicted probabilities*/
    %let zerorate = 0.9983;
    %let onerate = 0.0017;
    %let onesample = (344+&num_smote) / (199020 + 344 + &num_smote);
    %let zerosample = 1-&onesample;

    data casuser.smote_scored;
        set casuser.smote_scored;
        tempA = P_class1 / (&onesample / &onerate);
        tempB = P_class0 / (&zerosample / &zerorate);
        P_class1_adj = tempA / (tempA + tempB);
        P_class0_adj = tempB / (tempA + tempB);
        drop tempA tempB;
    run;

    /*plot Precision Recall Curve and display AUPRC*/
    proc logistic data=casuser.smote_scored(where=(_partind_=1)) noprint;
        model class(event="1")= / nofit outroc=work.smote_roc_logistic;
        roc pred=P_class1;
    run;

    title "Precision and Recall for SMOTE with &num_smote Synthetic Samples";
    %prcurve(data=work.smote_roc_logistic, options=optimal);
%mend;

The macro is left whole to make copying it easier, but the key steps are identified in the code comments and are summarized as follows:

1. The SMOTE Procedure is used to generate a user-specified (num_smote) number of synthetic fraud cases based on the original training data.
2. The newly created synthetic samples are merged back with the original training data to create the augmented training set. The FREQ Procedure is used to display the balance of fraud and non-fraud cases in the augmented training set.
3. The GRADBOOST Procedure is used with the default settings to train a gradient boosting model on the augmented training set.
4. The GRADBOOST Procedure is used to score the validation data using the model trained in the previous step.
5. An adjustment is made to the predicted probabilities generated by the gradient boosting model based on the addition of synthetic samples added to the data.
   1. The validation sample doesn’t contain synthetic fraud cases, so the percentage of fraud cases in the validation data doesn’t match the percentage of fraud cases in the augmented training set. The adjustment is a calibration of the predicted probabilities, making it easier to select a sensible probability cutoff.
   2. This is not necessary for analyzing the precision-recall curve or calculating the AUPRC but is a good step in general when working with datasets where we modify the balance of target and non-target events during the training process.
6. The LOGISTIC Procedure is used with the %prcurve() macro as described in previous posts to plot the precision-recall curve and calculate the AUPRC.

Now we try using the macro with a few different choices for the number of SMOTE samples added to the training data.

/*double the number of target events (one fake sample for every real sample)*/
%compare_auprc_smote(344);

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Recall from the previous post that the baseline model had an AUPRC of 0.8516, so although adding 344 synthetic samples yields a slightly better model, it is only a very small improvement in the AUPRC. Let’s see if a smaller number of synthetic samples helps improve the model performance.

/*add a small number of target events (one fake sample for every two real samples)*/
%compare_auprc_smote(172);

Using 172 synthetic samples yields a model with an AUPRC of 0.8688, which is somewhat better than the baseline performance of 0.8516.

/*add an even smaller number of target events (one fake sample for every four real samples)*/
%compare_auprc_smote(86);

When we only add 86 synthetic samples the AUPRC is 0.8590, which beats the baseline model but performs worse than the previous attempt with 172 synthetic samples.

/*does it even matter if we add so few target event? (one fake sample for every eight real samples)*/
%compare_auprc_smote(43);

Using 43 synthetic samples yields a worse model than using 86 or 172 synthetic samples, suggesting that we are not adding much useful information to the training data with this very small amount of synthetic data.

/*add 3000 target events (~10x increase)*/
%compare_auprc_smote(3000);

Adding 3,000 synthetic samples yields an AUPRC of 0.8618, which is an improvement above the baseline, but is still slightly lower than the AUPRC for the model with 172 synthetic samples. It’s hard to know exactly the ideal number of synthetic samples to add to the data, especially since it’s likely to be data-dependent. From this analysis it seems like adding a somewhat small number of synthetic samples is a better strategy than trying to balance the training data with a 50/50 split of fraud and non-fraud cases. This is just a simple experiment on a single rare target event dataset, so it is important to experiment with the number of synthetic samples when fitting models on your own rare target event data. This experiment was performed only comparing the models in terms of the addition of SMOTE samples, so improvement in the AUPRC could also be achieved by tuning the hyperparameters of the gradient boosting model, trying different machine learning models, or trying different data preparation techniques (other than adding synthetic samples).

/*add 30000 target events (~100x increase)*/
%compare_auprc_smote(30000);

Adding 30,000 synthetic samples starts to degrade model performance below the baseline model. At this point the synthetic samples overwhelm the real data and contribute negatively to the AUPRC. It’s challenging to establish a good threshold for the maximum number of synthetic samples to generate for any given rare target events dataset, but there is always a point where adding more synthetic samples starts to hurt the model performance rather than help it, and this number is often fewer synthetic samples that would be generated to balance the data to a 50/50 split between target events and non-target events.

The Area Under the ROC Curve uses numerical integration techniques like the trapezoidal rule to sum up the area of little boxes beneath the curve. We can do the same thing to calculate the Area under the Precision-Recall Curve (AUPRC), but this means we use an overly optimistic linear interpolation that is not valid in the Precision-Recall space. We find that changing the recall value does not necessarily create linear changes in the precision, evidenced in some of the plots above by the spiky nature of the Precision-Recall curve. A 2006 paper by Davis and Goadrich calculates a non-linear interpolation method that works in the Precision-Recall space, which allows us to add interpolated points to the curve and thus enables the use of the trapezoidal rule in calculating the AUPRC. This approach is used in the SAS %prcurve() macro described in previous posts and used in the experiments above.

Another important conclusion from the 2006 Davis and Goadrich paper is that algorithms that optimize the area under the ROC curve don’t necessarily optimize the AUPRC. This is true even though given any two curves the one dominant in the ROC space will also be dominant in the PRC space. This means that when working with rare target events model it is better to tune model hyperparameters based on the AUPRC on validation data rather than on traditional metrics like the ROC Index.

Another way to evaluate models based on the Precision-Recall curve is to calculate a metric closely related but not equal to the AUPRC called the Average Precision. This is an especially popular metric when working with machine learning models in Python because the scikit-learn package provides an easy function to calculate this (whereas the AUPRC requires the interpolation scheme described above). The Average Precision (AP) score is calculated as follows:

Average Precision = ∑_n(Recall_n - Recall_n-1) Precision_n

Where we are summing across the n different cutoff thresholds at which we calculate precision and recall. By default this is often 100 cutoff values from 0.00 to 1.00, but when using the ASSESS Procedure in SAS this bin size for calculating assessment statistics can be chosen manually. Unlike the AUPRC the AP calculation does not use interpolated values, so it does not require a special interpolation scheme.

The AP score can be calculated manually in SAS Viya using the ASSESS Procedure and the TSMODEL Procedure. We use the TSMODEL Procedure in SAS Viya for its ability to perform sequential array manipulations, in SAS 9 we can simply use a DATA Step to perform this calculation but the distributed nature of SAS Viya means that a DATA Step could yield unexpected results when performing sequential calculations with parallel worker nodes. The logic we use in the TSMODEL Procedure can easily be replicated in a SAS DATA Step for use in SAS 9.

The following PROC CAS Code calculates the AP score on a table Valid_Scored which contains predictions in the column P_class and true values in the column class. The ASSESS Procedure is used to calculate confusion matrix values (True Positives, TP; False Postives, FP; True Negatives, TN; and False Negatives, FN) across a range of cutoff thresholds, while the DATA Step is used to create an observation number variable to serve as a proxy time ID for the TSMODEL Procedure.

proc cas;
    action percentile.assess / 
        table='Valid_Scored'
        inputs='P_class'
        response='class'
        event='1'
        cutstep=.01
        casOut={name='valid_assess', replace=True};

    action dataStep.runCode / 
        code="data casuser.valid_assess_roc;
                  set casuser.valid_assess_roc;
                  obs = round(_cutoff_,0.01)*100+1;
              run;";

    action timedata.runtimecode / 
        table='valid_assess_roc'
        series={{name='_TP_'},{name='_FP_'},{name='_FN_'},{name='_TN_'},{name="_cutoff_"}}
        arrayOut={table={name='apcalc', replace=True},arrays={{name='precision'},{name='recall'},{name='AP_calc'}}}
        scalarOut={table={name='ap_scalar', replace=True}, scalars={{name='ap_sum'}}}
        interval="obs1"
        timeID={name='obs'}
        code="do i = 1 to _LENGTH_;
                  precision[i] = _TP_[i] / (_TP_[i] + _FP_[i]);
                  recall[i] = _TP_[i] / (_TP_[i] + _FN_[i]);
                  AP_calc[i] = (recall[i-1]-recall[i])*precision[i-1];
                  ap_sum += AP_calc[i];
              end;";  
quit;

The TSMODEL Procedure is used to calculate the value ap_sum and output it in the table ap_scalar containing the Average Precision score. Precision and recall are calculated from the TP, FP, and FN values at each cutoff value, and then the AP score is calculated from these precision and recall values at each cutoff threshold. It’s important to use the TSMODEL Procedure here in SAS Viya because we need to calculate a difference between subsequent recall values, and this information could become mixed up in a non-sequential distributed computing environment. Lagged values could be used in a DATA Step to do the same thing if we were working in SAS 9 or on a single machine environment.

In this post we explored ways to improve the AUPRC by changing the number of synthetic samples to generate using the SMOTE technique, but it’s important to remember that there are many other choices we can make (data preparation, hyperparameters, model selection) that can impact model performance, and we can try all of these techniques to improve the AUPRC for rare target event models. There are even other ways to generate synthetic data such as generative adversarial networks, and Bayesian approaches. We also discussed some details regarding interpolation when calculating the AUPRC (we must be careful of naively integrating the Precision-Recall curve), which emphasizes the value of the %prcurve() macro discussed in a previous post. Finally we introduce a similar metric, the Average Precision score, which is popular among Python users due to its availability in scikit-learn. There is no clear advantage to using AP over AUPRC when tuning rare target event models, although visualizing the Precision-Recall curve (a prerequisite for calculating the AUPRC) can be a helpful exercise when evaluating these models. The next post in this series will explore how we can use the solveBlackBox CAS Action from SAS Optimization to automatically tune hyperparameters in a gradient boosting model based on the AP score calculated in this post. This is a manual and code-based way to do autotuning, but it is also very flexible.

References:

• Machine Learning with a Rare Target Event Level – Part 1: Event-Based Sampling
• Machine Learning with a Rare Target Event Level – Part 2: Synthetic Minority Oversampling
• Machine Learning with a Rare Target Event Level – Part 3: Precision-Recall Curves
• Davis, J., & Goadrich, M. (2006, June). The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on Machine learning(pp. 233-240).
• Scikit-learn average_precision_score

Find more articles from SAS Global Enablement and Learning here.