How to do the model validation in SAS?

[buski]

Hi,

I would like to know how can I take an new dataset to validate an established model.

For example: If I have a model to investigate the relationships between health events and climatic variables in New York, how to use this model to fit the different region with the same variables?

I know it's easy to do in R but I have no idea how to do it in SAS.

Thanks a lot!

[Ksharp]

How about to use 'by region' statement in your proc to fit the different region?

Ksharp

[buski]

Thanks,

But I think "by region" will give me two set of different models...not model validation 

[Doc_Duke]

With logistic regression, you can fit an existing model to new data; the syntax is in the manual.  Then you can do something like a Hosmer-Lemeshow test to check the goodness of fit of the existing model to the new data.

Doc Muhlbaier

Duke

[buski]

Unfortunately, I used  "PROC PDL" function and the predprobs=x option does not apply.

[art297]

But can't you still use the STB option and proc score and determine how much variance you account for with each by variable?

[buski]

I don't think that use STB option and Proc SCORE is equivalent to the model validation.

I am not going to evaluate the importance of different variables.

What I want to do is to use a new dataset to validate the existing model.

[Ksharp]

Model validation is dependent on different Estimator.

Like F estimator for proc reg and AIC for other model.

I am not sure whether I understood your Model validation.

[Doc_Duke]

Buski,

I've not used PROC PDLREG, so I'm basing these comments on it's documentation.  It seems that PDLREG is linear regression with the lag variables from the time series included as an orthogonal polynomial.  If my understanding is correct, then a straightforward way to get a handle on the adequacy of the model for a new population is to compute the regression estimate (y-hat) on the new data using the existing model and then use that as a single covariate in a new model that is otherwise specified exactly as the original one was.  If the model is adequate, then the new coefficients will not be significant.  To the extent that they are significant (beyond that expected from type I randomness), there is evidence of inadequacy. 

This is not a nice clean single number as a "score", but it can be quite helpful in figuring out where the model needs work.

Doc Muhlbaier

Duke

[Rashu]

Buski, I was wondering if you figure out the way to validate a model. I am trying to do the same and having a hard time. I am trying to validate using ROC and calibration. I know how to get ROC on its own but I can't seem to figure out how to include the coefficients. Please let me know if you figured out. Thanks