topic Re: ROC curve: associate predicted probability with parameter values in Statistical Procedures

ROC curve: associate predicted probability with parameter values

deleted_user — Wed, 28 Jul 2010 11:04:08 GMT

Hi,
I'm using Proc Logistic to create a ROC curve; I have the Y variable which codifies the event of interest (cells differentiation, 1 = event = differentiation; 0 = non-event = non-differentiation) and the X independent variable which is a parameter measuring the differentiation level.

Here is the code:

proc logistic data=total descending noprint;
model Y=X / outroc=rocdata_sptot;
output out=sptot1 p=predprob;
run;

After that, I define a cut-off on the X to distinguish between events and non-events.

The correct interpretation of my data is: the higher the X, the higher the probability of differentiation (event=1); but the interpretation given by SAS is the opposite (i.e. the higher the X, the lower the probability of differentiation). I checked it in the out dataset, where a lower probability of event is associated to higher levels of my X.

Is there a statement or any other solution I can use to "reverse" the association done by SAS, in order to make it correctly fit my data?

Thanks a lot,
Anna

Re: ROC curve: associate predicted probability with parameter values

SteveDenham — Wed, 28 Jul 2010 12:10:19 GMT

Hi Anna,

Try this:

proc logistic data=total descending noprint;
model Y(event='1')=X / outroc=rocdata_sptot;
output out=sptot1 p=predprob;
run;

The way I read the documentation, this should give what you need.

Steve Denham

Re: ROC curve: associate predicted probability with parameter values

deleted_user — Wed, 28 Jul 2010 14:43:35 GMT

Dear Steve,
thank you very much for your answer.
I tried what you suggested but I still have the same results as before: SAS is still associating low values of predicted probability to high values of X, and I need the opposite.
I also tried with some variations but the results don't change.
Many thanks for your help anyway 🙂

Re: ROC curve: associate predicted probability with parameter values

Ksharp — Wed, 28 Jul 2010 16:00:28 GMT

then Don't use 'descending'.
But i am curious that Why you do want to do that?

Re: ROC curve: associate predicted probability with parameter values

deleted_user — Wed, 28 Jul 2010 19:00:07 GMT

I've already tried, I tried everything I can think of.
I need to do this because that's the way my data must be interpreted and handled, it's not for fun but because of the biological background 😉

Re: ROC curve: associate predicted probability with parameter values

Doc_Duke — Thu, 29 Jul 2010 01:57:52 GMT

I've got to think that this has something to do with the model or the data.

Have you done plots of the raw data? Maybe your putative biological model isn't reflected in the data. (SAS 9.2's ODS statistical graphics has some that are quite useful. Also, see Frank Harrell's book on regression modeling for diagnostics.)

Have you simplified the model for the presentation here? E.g. are there other covariables in the model? If so, and they are related to X, they can attenuate or even reverse the effect of X.

Doc Muhlbaier
Duke

Re: ROC curve: associate predicted probability with parameter values

Ksharp — Thu, 29 Jul 2010 10:45:21 GMT

Hi.Anna.
I think Doc@Duck has the right idea here.That is to say you need more covariates to add into your model or need more obs.Your logistic model is perfect prediction(i.e. you only have one outcome variable and one covariate).The perfect prediction is not a good model which will give your not good estimate and interpretation. I think SAS will issue a NOTE in the log file. Message was edited by: Ksharp

Re: ROC curve: associate predicted probability with parameter values

Doc_Duke — Thu, 29 Jul 2010 12:56:20 GMT

Ksharp mentioned perfect prediction. You won't get that unless there is total separation of the y's by the x's. We have, however, been assuming that X is a continuous variable. Although a logistic regression can be done with a single binary predictor, it is then just a Chi Square and the ROC is pretty meaningless.

Re: ROC curve: associate predicted probability with parameter values

SteveDenham — Thu, 29 Jul 2010 14:11:43 GMT

Anna,

If the descending option is kept, then at least try:

proc logistic data=total descending noprint;
model Y(event='0')=X / outroc=rocdata_sptot;
output out=sptot1 p=predprob;
run;

Good luck,

Steve Denham

Re: ROC curve: associate predicted probability with parameter values

Ksharp — Thu, 29 Jul 2010 15:35:31 GMT

Hi.Doc@Duke
I guess that the sample size is too small or the events are too sparse.So the estimation of coefficient would be bias.
So anna You can try to use exact logistic regression .Such as:

[pre]
proc logistic data=yourdataset descending exactonly;
model Y=X Z;
exact X Z /estimate=both;
run;
[/pre]

Ksharp

Re: ROC curve: associate predicted probability with parameter values

deleted_user — Mon, 02 Aug 2010 08:12:32 GMT

Dear All,
many thanks for your help!
I'm currently working on the problem, considering all your suggestions 🙂

Re: ROC curve: associate predicted probability with parameter values

deleted_user — Wed, 11 Aug 2010 14:20:37 GMT

Hi again,
I'm going crazy with this issue, again, I wasn't able to solve it in any way.
The problem is quite simple, I tried all the possible combinations but I can never find the right one.

** I am using proc logistic to build a ROC curve.
** I have a binary "event" variable (dependent). The two categories correspond to Differentiation and Undifferentiation.
** I have only 1 continuous predictor (independent)
** The right intrerpretation of the data is: the higher the predictor, the higher the differentiation

I tried all the combinations obtained changing the way of codifying the event (what it's coded as 0 and what as 1), and the event for which the probability of the logistic model is modeled. I will list them here:

1) Differentiated = 0
Undifferentiated = 1
Event for which probability is modeled = 1
------> RESULT: the higher the predictor, the higher the prob of UNdiffer --> WRONG

2) Differ = 0
Undiffer = 1
Event = 0
------> RESULT: the higher the predictor, the lower the probability of Differ --> WRONG

3) Differ = 1
Undiffer = 0
Event = 1
-------> RESULT: the higher the predictor, the lower the prob of Differ --> WRONG

4) Differ = 1
Undiff = 0
Event = 0
------> RESULT: the higher the predictor, the higher the prob of UNdiff --> WRONG

Do you think it's normal?
I'm convinced there must be something very wrong but I really don't know what to do.
Thanks again,
Anna

Re: ROC curve: associate predicted probability with parameter values

SPR — Thu, 07 Oct 2010 20:01:48 GMT

Hello Anna,

If you could supply a sample of you data to experiment?

SPR