topic Selecting model based on Average Profit with separate sampling in SAS Academy for Data Science

Selecting model based on Average Profit with separate sampling

pvareschi — Fri, 08 May 2020 15:35:12 GMT

Re: Applied Analytics Using SAS Enterprise Miner

At page 6-44 of the course notes, the example shows how to select a model based on Average Profit; however, are the values of Average Profit still biased when using separate sampling? In the example, only the prior probabilities have been adjusted, but the physical composition of the sample is still based on a 50/50 split: based on the formula given at page 6-43, I would expect that should result in a biased value for average profit. Or am I missing anything in the way Enterprise Miner handles prior probabilities/decisions behind the scenes?

The key question is whether we used a proportional sample would result in different values of Average Profit which may lead to choosing a different model compared to the approach based on an oversampled dataset.

Re: Selecting model based on Average Profit with separate sampling

gcjfernandez — Sat, 09 May 2020 18:58:06 GMT

Thank you for asking this question and for further clarification. When computing for average profit, SAS EM always adjust the frequencies of primary and secondary outcome events for prior probabilities when the prior probability adjustment is enabled in the decision setting (Please see page 6:24 course PDF). To illustrate this point I have modified the corresponding slide below:

Hope this explanation is adequate. If not please let me know.

Re: Selecting model based on Average Profit with separate sampling

pvareschi — Mon, 11 May 2020 07:14:56 GMT

Thank you for your clarification...and I like the amended slide!