How do I use the Model Comparison node with the Cutoff node results?

[cdettmann0]

Hi,

I am trying to use the Cutoff node in SAS Enterprise Miner 14.1 (university "cloud" version) for a logistic regression.  I then want to compare the results of changing the cutoff to the results I get from three other (similar) logistic regressions that don't use the cutoff, just to see the differences in model performance.

When I connect my four models (three without the Cutoff and one with) to the Model Comparison node, the results don't seem to capture the model that uses the cutoff.  It produces results, but they appear to be the same as I got before the cutoff (yes, I changed the user input cutoff value from 0.5 to 0.03).

Am I not using the Model Comparison correctly with the Cutoff node?  I'm fairly new with SAS, so I'm kinda struggling with this one and the examples I've found seem to stop short of comparing models.

Thanks in advance.

[cdettmann0]

PS:  I forgot to mention I have a 65% training and a 35% testing set (from the "Data Partition") node.

[WendyCzika · Accepted solution]

Currently in EM, the Model Comparison node does not use the cutoff from the Cutoff node; however many of the assessment statistics are not affected by the cutoff.  Misclassification rate is of course, but the ASE, lift and gain-related statistics, and KS are not so those wouldn't change based on the choice of cutoff anyway.

[cdettmann0]

Ok, thanks Wendy, that actually makes sense, I think.  In other words, it's not changing the equation of the predictive line...just the point at which the model chooses to assign a 1 or a 0 to the observation, correct?  I think I just expected to see a separate row in the fit statistics, but I suppose that could be considered redundant.  Appreciate the reply!