Hi Honest_Abe,
On the page where the instructions call out the edits to the "Prior Probabilities" tab, there's other instructions around making changes on the "Decision Weights" tab as well. If you made those changes, then the model assessment criterion is based on those decision weights. If you use the numbers in the book, that means that each row in your data set represents a 25% chance of making $14.50, and a 75% chance of losing $0.50 (or making -$0.50, equivalently). Think about betting $0.50 to win a $15.00 jackpot.
If your probability of winning that bet (in the population, no models, no predictions) was 25%, then you should probably take that bet all day, every day! Your expected "winnings" are (.25 x $14.5) + (.75 x -$0.50) = $3.25. Don't build a model, just take that bet 🙂 [This is what I suspect happened in your case. Enterprise Miner built a series of trees and a series of regressions, but none of them could beat this average profit figure, so it "Occam's Razor"-ed you and took the simplest model that gave the best results. It's hard to get a simpler model than "mail everybody, all of the time," so that's what the tree and the regression gave you.
But! What if you lived in a world where the baseline "success" rate (the probability that TARGET_B=1) is closer to 5% than 25%? Then for each trial, your expected profit is (0.05 x $14.5)+(0.95 x -$0.50) = $0.25. Now we're talking about betting $0.50 to try and win $0.25. (In many real world cases, the expectation is negative, and you're worse off than that.) So how do we gain an advantage? Build a model, and target the sub-population that has a favorable expected value: everyone gets a predicted probability (p), and you can choose to mail only if (p x $14.5)+[(1-p) x -$0.50] comes out to be "large enough." "Large enough" could mean "positive," or it could be subject to some other constraints/considerations.
If you don't know good numbers for those profit/loss values, or this expected-value argument doesn't meet your needs, then you can actually tell the model nodes to select the best model according to validation data average squared error and you'll probably get results more in line with your expectations.
If you didn't put any values in the "Decision Weight" tab, then I need to re-visit your question. If you have any other details about steps you were experimenting with, that might be a good clue.
Thanks!
... View more