topic Re: Plotting Calibration curve/line in Graphics Programming

Plotting Calibration curve/line

Rashu — Tue, 02 Oct 2012 03:23:26 GMT

Can anyone help me on how to plot a calibration curve/line with binary outcome? When I plot the predicted probability vs. the actual outcome I get straight line thru 0 and 1 because of binary outcome. I can't seem to figure out otherwise.

Thanks

Re: Plotting Calibration curve/line

Rick_SAS — Tue, 02 Oct 2012 12:15:48 GMT

I assume that you have one or more continuous explanatory variables?

I usually use PROC LOGISTIC to model the data and use the PLOTS=EFFECT statement or the newer EFFECTPLOT statement to graph the results:

http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sect060.htm#statug.logistic.tlog3f

If you prefer to do it "by hand," plot the predicted probabilities as Y and the explanatory variable as X.

Re: Plotting Calibration curve/line

Rashu — Wed, 03 Oct 2012 23:00:42 GMT

Rick,

Thank you for you comment. I do have explanatory variables. However, I am not trying to plot the explanatory variable to predicted probability which is what EFFECTPLOT gives. What I am trying to do is plot the predicted probability versus the actual outcome. I already have a model (formula) that I can calculate the predicted probability with but my outcome is binary. So when I plot the predicted versus actual outcome, I get two lines thru o and 1 for binary. I think I need to divided the prob in deciles and plot against the actual frequency and I can seem to figure out how. Please let me know if this makes sense. Thanks for the help.

Re: Plotting Calibration curve/line

Rick_SAS — Wed, 03 Oct 2012 23:28:05 GMT

Unfortunately I am not understanding what you are trying to do. Can you link to a web page that has a picture of what you are trying to achieve?

Re: Plotting Calibration curve/line

Rashu — Thu, 04 Oct 2012 02:16:12 GMT

Here is the link of the article. It is similar to what I am trying to do. Figure one is what I am trying to get. Thank you.

Re: Plotting Calibration curve/line

Rick_SAS — Thu, 04 Oct 2012 09:38:16 GMT

The link didn't post.

Re: Plotting Calibration curve/line

Rashu — Thu, 04 Oct 2012 13:19:39 GMT

http://igitur-archive.library.uu.nl/med/2006-0906-200629/grobbee_05_assessingtheapplicabilityofscoringsystems.pdf

Re: Plotting Calibration curve/line

Rick_SAS — Thu, 04 Oct 2012 16:19:47 GMT

OK. That's clearer. This isn't really a "graph" question, it's a how do I compute the quantities needed for a graph" question.

Here's what you need:
1) Compute the deciles of the predicted probabilities

2) For each decile, compute the mean and upper/lower 95% confidence interval for the observed outcome. The mean is also the "percentage of observed values that are 1."

A "short answer" is that you can do it like this:

/* set deciles of predicted risk */

data Deciles;

set Pred; /* data that includes variable PredProb for predicted probabilities */

Decile = int(10*PredProb)/10;

run;

proc sgplot data=all;

dot decile / response=y stat=mean limitstat=clm;

run;

This will get you in the ballpark, and would be sufficient for "internal" plots that you intend for yourself or your group.

Unfortunately there are three problems with this approach if you are trying to EXACTLY reproduce the figure in the paper:

1) The DOT statement displays a graph with the deciles on the vertical axis, which is opposite from the graph in the paper.

2) The LIMITSTAT= option computes confidence limits by using the standard formula for normally distributed data. These data are binary, and therefore you should really use CIs for binomial proportions (not a big deal if you have lots of data, but still...)

3) The plot in the paper also overlays a curve which I assume is a nonparametric smoother (for example, a loess curve) through the (Y, PredProb) points.

All of these problems can be surmounted: call PROC FREQ to get the stats and then overlay the SCATTERPLOT / YUPPERLIMIT= YLOWERLIMIT= statement with a LOESS curve.

Re: Plotting Calibration curve/line

Rick_SAS — Thu, 04 Oct 2012 16:32:46 GMT

Oh, and properly you should use PROC RANK to get the deciles instead of the quick approximation that I used here.

Re: Plotting Calibration curve/line

bioning — Mon, 20 Feb 2017 01:17:56 GMT

Dear Rick:

Sorry to bother you! Would you please clarify a couple of points regarding the three differences, as you mentioned, between the graph of interest (Figure 1 in the attachment) and the graph you drew?

1)How would you going to fix the problem of deciles being on the vertical axis instead of the horizontal axis;

2)How exactly would you implement the "overlay the SCATTERPLOT / YUPPERLIMIT= YLOWERLIMIT= statement with a LOESS curve".

I understand that these codes may be super easy to you, but they are actually the bottleneck for me for this question. I googled for one day and found nothing. Thank you!

Re: Plotting Calibration curve/line

Rick_SAS — Mon, 20 Feb 2017 13:18:45 GMT

If you post sample data, we can make concrete suggestions. But it sounds like you want something like the following. Here I am using PROC SGPLOT, which has a simple syntax:

data Have;
input decile y low hi;
datalines;
1 1 0 2
2 3 1 2
3 4 2 5
4 6 4 7
5 5.5 5 6
6 5 4 6
7 4.5 4 7
8 3 1 5
9 2 1 3
;
proc sgplot data=Have;
scatter x=decile y=Y / YErrorLower=low YErrorUpper=hi;
loess x=decile y=y;
run;

Re: Plotting Calibration curve/line

bioning — Mon, 20 Feb 2017 17:03:04 GMT

Dear Rick:

Thank you so much for the instructions!

Now I'm just one step away from the figure in that article. Below are the codes I learned from you and applied to this question. The dataset was attached in the attachment. The variable phat_mean is the predicted risk by group, and the ob_risk is the observed risk by group. The 10 groups were divided based on the deciles of the predicted risk. My final question is: how to reproduce that dashed diagonal line, which appears to be a reference line, in Figure 1 of that article?

proc sgplot data=ning;
scatter x=phat_mean y=ob_risk / YErrorLower=Lower_CI YErrorUpper=Upper_CI;
loess x=phat_mean y=ob_risk;
run;

Looking forward to your further instruction!

Many thanks!

Ning

Re: Plotting Calibration curve/line

Rick_SAS — Mon, 20 Feb 2017 18:01:05 GMT

Glad you are making progress. I suggest you also add the NOMARKERS option to the LOESS statement. Then use the LINEPARM statement, like this:

proc sgplot data=ning;
scatter x=phat_mean y=ob_risk / YErrorLower=Lower_CI YErrorUpper=Upper_CI;
loess x=phat_mean y=ob_risk / nomarkers;
lineparm x=0 y=0 slope=1 / lineattrs=(pattern=dashed);
run;

Re: Plotting Calibration curve/line

bioning — Mon, 20 Feb 2017 23:42:39 GMT

Dear Rick:

Thank you for your tremendous help!

This "calibration plot" is an question in my assignment of a master-degree university course. To the best of my knowledge, your solution is the only, and THE BEST, resource that is available online as of now. In my homework, I cited your instructions and wrote: "The solution and the codes were developed under the guidance of, and virtually by, Mr. Rick from the SAS Institute. Web link listed below. https://communities.sas.com/t5/forums/replypage/board-id/sas_graph/message-id/11641"

The final graph is enclosed in the attachment for other SAS users' reference. This calibration plot is widely used to illustrate the performance of a "risk prediction model" in the medical land. It compares the predicted risk with the observed risk by level of the predicted risk. Statistically, it is also a visualization of the "Hosmer-Lemeshow test" that examines the extent to which the predicted values produced from the statistical model match the observed values obtained from the real world.

Many thanks again!

Ning

Re: Plotting Calibration curve/line

Rick_SAS — Sat, 21 Jul 2018 11:25:09 GMT

For future reference, here are two blog posts about calibration curves in SAS:

Calibration plots in SAS [using loess curves]

Decile calibration plots in SAS

The second article includes a comparison of the two methods. It ends with the following recommendation:

"Many leading researchers in logistic regression do not recommend the Hosmer-Lemeshow test for these reasons. The decile-based calibration curve shares the same drawbacks. Since SAS can easily create the loess-based calibration curve (see the previous article), there seems to be little reason to prefer the decile-based version."

Re: Plotting Calibration curve/line

rajdesai223 — Sun, 19 Aug 2018 03:39:46 GMT

Hi,

Do you mind posting the entire code from the start please? I don't understand what is phat_mean? Is it the mean of the predicted probability? And what is ob_risk ?

Thanks

Re: Plotting Calibration curve/line

Rick_SAS — Sun, 19 Aug 2018 10:34:02 GMT

The two blog posts that I linked to have complete code for analyzing a simulated data set.

Re: Plotting Calibration curve/line

rajdesai223 — Sun, 19 Aug 2018 15:09:03 GMT

Hi Rick,

I have a binary outcome variable called adherence(codes as 0 and 1). I ran the codes you suggested.

Here is my code.

proc logistic data=cases plots=effect;
class sex1 race group age1 hmo ER FmsScore;
model adherence(event="0")=sex1 race group age1 hmo ER FmsScore;
output out=LogiOut predicted=PredProb; /* save predicted probabilities in data set */
run;

proc rank data=LogiOut out=LogiDecile groups=10;
var PredProb;
ranks Decile;
run;

/* Then compute the mean predicted prob and the empirical proportions (and CI) for each decile */
proc means data=LogiDecile noprint;
class Decile;
types Decile;
var adherence PredProb;
output out=file mean=obs_risk PredProbMean
lclm=Lower_CI uclm=Upper_CI;
run;

proc sgplot data=file noautolegend aspect=1;
lineparm x=0 y=0 slope=1 / lineattrs=(color=grey pattern=dash);
loess x=PredProbMean y=obs_risk; /* if you want a smoother based on deciles */
scatter x=PredProbMean y=ob_risk / yerrorlower=Lower_CI yerrorupper=Upper_CI;
yaxis label="Observed Probability of Outcome";
xaxis label="Predicted Probability of Outcome";
run;

Here are the issues I am facing(I have attached a csv file for you)

1) There are unequal number of observations in each decile

2) my observed risk and predicted risk are very different in each group because of which I am not getting the plot correct.

Please help.

Thank You again.

Re: Plotting Calibration curve/line

Rick_SAS — Sun, 19 Aug 2018 18:43:30 GMT

The main issue is that you are modeling

adherence(event="0")

whereas in my blog I model adherence(event="1"). Thus your graph does not fall along the line y=x.

That's not an important concern. You can either model adherence(event="1") or you can change the LINEPARM statement to conform to the situation that you have, as follows. (Note that I changed some variable names to conform to the names in your CVS file.):

proc sgplot data=A noautolegend aspect=1;
   lineparm x=0 y=1 slope=-1 / lineattrs=(color=grey pattern=dash) clip;
   loess x=PredProbMean y=obs_risk;
   scatter x=PredProbMean y=obs_risk / yerrorlower=yLower yerrorupper=yUpper;
   yaxis label="Observed Probability of Outcome";
   xaxis label="Predicted Probability of Outcome";
run;

Re: Plotting Calibration curve/line

rajdesai223 — Mon, 20 Aug 2018 13:37:46 GMT

Thank You Rick. This was very helpful.