topic Re: ROC curve not always showing up in model comparison in SAS Data Science

ROC curve not always showing up in model comparison

Joel_Wesley — Thu, 07 Aug 2014 15:29:59 GMT

Hello,

The ROC curve (and various other output) aren't always showing up when I open the results to the model comparison node. Instead I only get Fit Statistics and Output. Occasionally it shows up, but I can't seem to figure out why it does in some diagrams/flows but not in others. Has anyone else had the same problem? I'm working in EM 12.1.

Thanks!

Re: ROC curve not always showing up in model comparison

WendyCzika — Thu, 07 Aug 2014 16:36:19 GMT

When it's not showing up, do you have a binary target and the property ROC Chart set to Yes?

Re: ROC curve not always showing up in model comparison

Joel_Wesley — Thu, 07 Aug 2014 16:42:21 GMT

Yes, those are always my settings. I'm beginning to think that EM simply won't calculate ROC when the sample becomes too large. I suppose that makes sense, as it involves running through the data repeatedly with many different cutoffs. Still, I have obtained ROC curves in base SAS with far larger samples. This won't seem to do it with ~3000.

Re: ROC curve not always showing up in model comparison

WendyCzika — Thu, 07 Aug 2014 16:59:09 GMT

Hmm, there are no sample size limitations that I'm aware of. This sounds like one for SAS Tech Support. Can you please create a model package with the flow where you are seeing this issue (see 46764 - Creating a model package (SPK file) in SAS® Enterprise Miner(tm)) and send it to Tech Support for assistance: http://support.sas.com/ctx/supportform/createForm.

Re: ROC curve not always showing up in model comparison

Joel_Wesley — Thu, 07 Aug 2014 17:37:31 GMT

Sounds good. Thanks for your help!

Re: ROC curve not always showing up in model comparison

YujunChiang — Fri, 08 Aug 2014 06:21:25 GMT

perhaps you only need to set the role of target variable from "interval" to "binary". I found that when target variable has value 0 and 1, the default role is not "binary", and I can't see roc before I set the role to "binary".

Re: ROC curve not always showing up in model comparison

Joel_Wesley — Fri, 08 Aug 2014 13:30:13 GMT

Thanks, it is set it to binary, but I still find it not producing ROC curves.

Re: ROC curve not always showing up in model comparison

Joel_Wesley — Wed, 29 Oct 2014 16:32:14 GMT

Just to update this thread - I received help a while ago from SAS tech support. Seems as though I shouldn't have been including segment variables - i.e., I should have been "rejecting" them at the data loading stage. I was keeping them, but calling them "segment." SAS tech also helped me use the "advanced" editor on data loading stages, which I wasn't trying before. Anyway, problem solved - ROC curves are showing up consistently now!

Re: ROC curve not always showing up in model comparison

M_Maldonado — Wed, 29 Oct 2014 17:34:58 GMT

Very useful update Joel!

Thank you!

Re: ROC curve not always showing up in model comparison

bcastillo10 — Sun, 23 Oct 2016 20:12:30 GMT

Thanks for the details and the update.