Calibration Curve / Stats + PPV / NPV

[CraigSmith]

Hi all,

Hope you're well - was wondering if there's a simple way to get calibration curve + metrics + PPV /NPV for my logistic regression model (code + pic below) ? Hopefully a simple addition of code?

Best wishes, 

PROC LOGISTIC DATA = ARCAGE;
FORMAT SMOK SMOK. DRINK DRINK. SEX SEX. PROLEV PROLEV. ALCOIC ALCOIC. TEETH TEETH. ;
CLASS SMOK ( REF = 'Never smoker')
DRINK (REF = '0: Never')
SEX ( REF = '2: Female' )
PROLEV (REF = '3: University degree/Manager')
ALCOIC (REF = '3: 3-4')
TEETH ( REF = '0: Never');

model CASE = AGE10 SEX PROLEV
SMOK SMOKP
DRINK ALCOIC
TEETH
/rocci RL SELECTION = F;
run;

[Quentin]

For calibration plots, see:

https://blogs.sas.com/content/iml/2019/02/20/easier-calibration-plot-sas.html

I wouldn't think you could get PPV/NPV from proc logistic, because you would need a way to tell it a cutpoint for deciding if the test/prediction is positive or negative (based on the predicted probability).

But I would think you could output the predicted probabilities from the model, then use a data step  to use whatever probability cutpoint you want to calculate postive/negative.  Then you could use PROC FREQ to get Sens/Spec/PPV/NPV.

[CraigSmith]

Thanks Quentin,

Sorry had some tech issues. Shall give this a crack and report back.

KR,

Craig 

[CraigSmith]

Hi Quentin,

Thanks for your answers - hope you're well. Have tried the calibration code but unsure how to make it work to my model - do you know what I'm doing wrong. 

Best wishes,

Craig 

/* NEW in SAS/STAT 15.1 (SAS 9.4M6): PLOTS=CALIBRATION option in PROC LOGISTIC */
title "Calibration Plot for a Quadratic Model";
title2 "Created by PROC LOGISTIC";
proc logistic data=LogiSim plots=calibration(CLM ShowObs);
model y(Event='1') = x x*x / GOF; /* New in 15.1: More goodness-of-fit statistics */
run;

PROC LOGISTIC DATA = ARCAGE;
plots=calibration(CLM ShowObs);
model y(Event='1') = x x*x / GOF; /* New in 15.1: More goodness-of-fit statistics */
run;

[Quentin]

Hi,

I'm not familiar with calibration curves.  But can you describe what is going wrong with your attempt?  Are you getting errors in the log?  (Please post log)  Or are you getting unexpected results?  

-Q.

[CraigSmith]

Hi Quentin,

Tried adapting code from https://blogs.sas.com/content/iml/2019/02/20/easier-calibration-plot-sas.html

to my data / logistic model 

/* NEW in SAS/STAT 15.1 (SAS 9.4M6): PLOTS=CALIBRATION option in PROC LOGISTIC */
title "Calibration Plot for a Quadratic Model";
title2 "Created by PROC LOGISTIC";
proc logistic data=ARCAGE plots=calibration(CLM ShowObs);
model y(Event='1') = x x*x / GOF; /* New in 15.1: More goodness-of-fit statistics */
run;

Gave log error of 

3547 /* NEW in SAS/STAT 15.1 (SAS 9.4M6): PLOTS=CALIBRATION
3547! option in PROC LOGISTIC */
3548 title "Calibration Plot for a Quadratic Model";
3549 title2 "Created by PROC LOGISTIC";
NOTE: PROCEDURE LOGISTIC used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds

ERROR: Variable Y not found.
NOTE: The SAS System stopped processing this step because of
errors.
3550 proc logistic data=ARCAGE plots=calibration(CLM ShowObs);
3551 model y(Event='1') = x x*x / GOF; /* New in 15.1:
3551! More goodness-of-fit statistics */
3552 run;

I'm essentially unsure how to adapt the code from that to mine (zero coding background) thought DATA = ARCAGE would fix it but apparently not. 

[Quentin]

Hi,

If there is a SAS programmer nearby, you might want to ask them for help.

Note that your original model statement referred to variables that exist in your dataset ARCAGE.

The new model statement you tried is: 

model y(Event='1') = x x*x / GOF; 

But your dataset ARCAGE does not have variables named Y or X.    That causes the key ERROR message in the log:

ERROR: Variable Y not found.

You could try changing back to your original model statement, or a simpler version of that model statement.

[CraigSmith]

Thanks Quentin,

Hmmm I'm without any stats support in my department till the 20th.

Was hoping calibration curve would be relatively straightforward to do myself.

Shall keep code form forums and see if they can modify it. 

BW,

Craig  

[Ksharp]

https://blogs.sas.com/content/iml/2018/05/14/calibration-plots-in-sas.html

https://blogs.sas.com/content/iml/2018/05/16/decile-calibration-plots-sas.html

https://blogs.sas.com/content/iml/2019/02/20/easier-calibration-plot-sas.html