Selecting model based on Average Profit with separate sampling

[pvareschi]

Re: Applied Analytics Using SAS Enterprise Miner

At page 6-44 of the course notes, the example shows how to select a model based on Average Profit; however, are the values of Average Profit still biased when using separate sampling? In the example, only the prior probabilities have been adjusted, but the physical composition of the sample is still based on a 50/50 split: based on the formula given at page 6-43, I would expect that should result in a biased value for average profit. Or am I missing anything in the way Enterprise Miner handles prior probabilities/decisions behind the scenes?

The key question is whether we used a proportional sample would result in different values of Average Profit which may lead to choosing a different model compared to the approach based on an oversampled dataset.

[gcjfernandez · Accepted solution]

Thank you for asking this question and for further clarification. When computing for average profit, SAS EM always adjust the frequencies of primary and secondary outcome events for prior probabilities  when the prior probability adjustment is enabled in the decision setting (Please see page 6:24 course PDF). To illustrate this point I have modified the corresponding slide below:

Hope this explanation is adequate. If not please let me know.

[pvareschi]

Thank you for your clarification...and I like the amended slide!