topic Re: How can I perform principal component analysis for logistic regression via SAS? in Statistical Procedures

How can I perform principal component analysis for logistic regression via SAS?

Season — Sat, 25 Feb 2023 02:22:06 GMT

I am currently building a logistic regression model whose dependent variable follows a binomial distribution. Based upon my professional knowledge, I assume that collinearity exists among the independent variables. Therefore, I wish to perform principal component analysis to detect possible collinearities and to lower the dimension of the independent variables. How can I do this via SAS? Thanks!

Re: How can I perform principal component analysis for logistic regression via SAS?

PaigeMiller — Sat, 25 Feb 2023 11:22:44 GMT

PROC PRINCOMP will do this.

It will find reduced dimensions you can use, but CAUTION: some of those reduced dimensions may not be good predictors.

A better procedure, in my mind, is Logistic Partial Least Squares regression, which will find reduced dimensions that are good predictors (as good as the data will allow). While (non-logistic) Partial Least Squares regression is available in PROC PLS, Logistic Partial Least Squares is not available in SAS but is available as a package in R.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Sat, 25 Feb 2023 12:32:08 GMT

Thank you very, very much, Paige, for your kind help! Actually, I have not been that familiar with principal component analysis as well as PROC PRINCOMP. Therefore, I previously thought that PROC PRINCOMP only supports principal component analysis for models whose independent variable is a continuous one.

One issue that bothers me much is the lack of information on how to perform principal component analysis for logistic regression via SAS. Since SAS Help has not provided an example on how to perform principal component analysis for logistic regression and I retrieved no results for my question after browsing SAS Community Library, could you please provide some hint on the detailed procedure of doing so? Or perhaps a tutorial written by someone else?

Many thanks!

Re: How can I perform principal component analysis for logistic regression via SAS?

PaigeMiller — Sat, 25 Feb 2023 12:52:26 GMT

@Season wrote:

Thank you very, very much, Paige, for your kind help! Actually, I have not been that familiar with principal component analysis as well as PROC PRINCOMP. Therefore, I previously thought that PROC PRINCOMP only supports principal component analysis for models whose independent variable is a continuous one.

Principal components does not use a Y-variable. Therefore, you can use it on the X-variables with either continuous Y-variables or categorical Y-variables, it doesn't matter.

One issue that bothers me much is the lack of information on how to perform principal component analysis for logistic regression via SAS. Since SAS Help has not provided an example on how to perform principal component analysis for logistic regression and I retrieved no results for my question after browsing SAS Community Library, could you please provide some hint on the detailed procedure of doing so? Or perhaps a tutorial written by someone else?

It is no different than performing Principal Components for continuous Y. The Y-variable(s) are simply not used by PCA. As I stated above, (some of) the dimensions it finds may not be good predictors of Y.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Sat, 25 Feb 2023 13:16:57 GMT

Ok, thank you very much for your help. Actually, the current model I have built works not bad. Still, some of the parameters that have been proved to be associated with the independent variables by professional knowledge have been tested as statistically insignificant in my analysis. Therefore, for the sake of improving my model, I have come to seek help to examine if the insignificances were caused by collinearities, by the lack of samples, or by other issues (e.g. outliers).

You have repeated reminded me that in the circumstance I am consulting, principal component analysis may not be the best choice. Thank you for your reminder. Actually, I have only systematically studied statistics and the mathematical knowledge it bases upon for an entire year. Therefore, I can only use SAS right now. I will try Logistic Partial Least Squares method if principal component analysis failed to tackle this problem.

Thank you very much again!

Re: How can I perform principal component analysis for logistic regression via SAS?

StatDave — Sat, 25 Feb 2023 16:23:02 GMT

Another thing to consider is a penalty-based model selection process such as LASSO which is available in PROC HPGENSELECT and selects a subset of the candidate predictors rather than combine them all into a small number of functions. Also, note that if the concern is more about collinearity causing ill-conditioning of the information matrix used in the model-fitting process than dimension reduction of your predictors, then that can be addressed as discussed in this note.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Sun, 26 Feb 2023 03:28:33 GMT

Thank you for the help you have offered! Currently, my most important concern is the diagnostics of collinearity. I will take a look at LASSO and the note you have mentioned.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Sun, 26 Feb 2023 03:50:38 GMT

Oh, by the way, I have another problem concerning the diagnostics (discovery) of collinearities in logistic regression. In linear regression models, tolerance, variance inflation factor (VIF), as well as condition index (computed from eigenvalues) can serve as indicators of collinearities among the independent variables in the model. The aforementioned three statistics can be computed in PROC REG upon request. However, they are not available in the modules that build logistic regression models (i.e. PROC LOGISTIC, PROC GENMOD, PROC HPLOGISTIC, etc.). Therefore, diagnostics of collinearity in logistic regression is not that easy.

I tried PROC PRINCOMP in my data today and found out that PROC PRINCOMP does not compute the three statistics either. Instead, it produces a correlation matrix of the variables I wish to analyze. There is no surprise that "strong" correlations exist among the variables I put in the logistic regression model, with some of the correlation coefficient reaching 0.6154. I guess that collinearities must exist in this situation.

So here are my questions: when it comes to diagnostics of collinearity, can correlation coefficients serve as surrogate statistics for tolerance, VIF and condition index in logistic regression? If not, what statistic(s) can do this job? Also, how can I compute tolerance, VIF and condition index in logistic regression?

Could @PaigeMiller, @StatDave or someone else kindly give me a hand?

Thank you all very much!

Re: How can I perform principal component analysis for logistic regression via SAS?

StatDave — Sun, 26 Feb 2023 04:01:37 GMT

The note I referred you to in my last post specifically discusses and shows how to get collinearity diagnostics for a logistic (or other generalized linear model). I suggest you read the collinearity section of that note and use the method shown. As noted there, correlation among your predictors by themselves is not necessarily a problem. But as I also mentioned, you might not even need to bother with the diagnostics if you use the penalty-based LASSO selection method to just pick out the important predictors.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Sun, 26 Feb 2023 04:39:24 GMT

OK, thank you very much for your help! I will read the note you have mentioned carefully and try LASSO as well to compare the two methods. It's too bad that SAS Community only supports accepting merely one reply as the solution. I think that your replies and the replies given by @PaigeMiller are all very fruitful for not only me, but also all of those that are troubled by collinearity in logistic regression. After all, I have retrieved nearly zero article discussing the solution of the collinearity problem in my search for articles on the Internet. Instead of discussing much about mathematical or statistical theories prior to providing a solution (like most articles do), your replies get straight to the point-- provide answers to the problem directly. I myself deem your replies as wonderful "concise textbooks" to the problem. I am sure that your replies can benefit other researchers who are struggling to find a solution to that problem and spending much time on searching for information instead of data analysis itself.

By the way, I major in medicine and is familiar with a few search engines that specialize in searching for articles on medicine (e.g. PubMed). Could you please introduce the search engine statisticians frequently use (aside from Google Scholar) or a few prestigious journals on statistics?

Thank you both for your kind help again!

Re: How can I perform principal component analysis for logistic regression via SAS?

PaigeMiller — Sun, 26 Feb 2023 11:45:24 GMT

To use the VIF in PROC REG, you create a made up variable that is a continuous Y and use your X-variables. The VIF does not depend on the Y variable.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Sun, 26 Feb 2023 12:10:04 GMT

OK, I see. Computing VIF in PROC REG when the dependent variable is a continuous one is easy. Yet the question I raised earlier is the computation of VIF in a logistic regression model. Can SAS do that? Thanks!

Re: How can I perform principal component analysis for logistic regression via SAS?

StatDave — Sun, 26 Feb 2023 16:09:05 GMT

If you read through the note I referred to then you should have learned that collinearity diagnostics (like VIF) for a logistic model (or any generalized linear model) requires using appropriate weights in PROC REG. As is specifically illustrated for a logistic model in that note, the weights can be obtained by first fitting the model in PROC GENMOD and saving the HESSWGT= values. When you then fit the model (to any response values) in PROC REG using those weights, you get the appropriate collinearity diagnostics for assessing your logistic model.

Re: How can I perform principal component analysis for logistic regression via SAS?

PaigeMiller — Sun, 26 Feb 2023 23:15:12 GMT

@Season wrote:

OK, I see. Computing VIF in PROC REG when the dependent variable is a continuous one is easy. Yet the question I raised earlier is the computation of VIF in a logistic regression model. Can SAS do that? Thanks!

No you don't see. I said: "To use the VIF in PROC REG, you create a made up variable that is a continuous Y and use your X-variables. The VIF does not depend on the Y variable."

And of course, there's also the point made by @StatDave

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Mon, 27 Feb 2023 15:23:04 GMT

Thank you for your kind and repetitive reminder. In fact, I had just begun reading the note you mentioned when I was replying to @PaigeMiller yesterday. I am now fully informed of the fact that weights should be multiplied when it comes to diagnosing collinearity in generalized linear models.

Still, I have some questions:

(1) I noticed that the var argument of PROC STANDARD standardizes all of the independent variables in the logistic model (li, temp and cell). Now that collinearity exists only between variable temp and the intercept, does all of the independent variables have to be standardized?

(2) The means of obliterating (or at least reducing) collinearity in a logistic regression model demonstrated here is variable standardization. In a complete model building process, what follows the PROC STANDARD procedure is using these standardized variables to perform logistic regression modeling. Eventually, the user may wish to transform the standardized variables into unstandardized ones. When I was a student studying statistics, my teacher demonstrated an example of using SAS to perform principal component analysis for multivariate linear regression. She completed the final process (i.e. transform the standardized variables back to the unstandardized ones after the entire model building process) by writing down the equation in hand and perform arithmetic calculations on her own.

Is there an automatic way of doing that final transformation process by SAS?

(3) The circumstance illustrated in the note you provided was one where one independent variable collinears with the intercept. What if the independent variables collinear with each other? Aside from deviating from the original model (i.e. switching to penalty-based model selection process like LASSO or other methods like Logistic Partial Least Squares Regression, etc.) and simply deleting one or more variables involved in collinearity, is variable standardization still a solution to that problem? If so, should the researcher standardize all the independent variables, as is the case in the note you provided; or just the independent variables that are involved in collinearity?

Many thanks!

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Mon, 27 Feb 2023 15:12:02 GMT

Oh, yes, you were totally correct (laugh oh laugh), I surely did not see yesterday.

@PaigeMiller wrote:

@Season wrote:

OK, I see. Computing VIF in PROC REG when the dependent variable is a continuous one is easy. Yet the question I raised earlier is the computation of VIF in a logistic regression model. Can SAS do that? Thanks!

No you don't see. I said: "To use the VIF in PROC REG, you create a made up variable that is a continuous Y and use your X-variables. The VIF does not depend on the Y variable."

I did not completely understand what you mean yesterday and was merely focusing one the word "continuous" before the letter "Y". I thought you must have misunderstood me, since I made a lengthy reply yesterday, with my questions hidden between the lines. As a result, you may had just skimmed through my reply, without noticing that I was modeling a discrete independent variable. So that's the underlying reason for my further reply you quoted.

And of course, there's also the point made by @StatDave

Now I think that I have truly seen what you meant. But other questions emerged when I was reading the note @StatDave cited. I have already raised my questions in my latest reply to @StatDave. I wonder if you could offer your suggestion to the three questions, if you don't mind.

Thank you very much for your patience and your time spent on my questions again!

Re: How can I perform principal component analysis for logistic regression via SAS?

PaigeMiller — Mon, 27 Feb 2023 15:53:50 GMT

@Season wrote:

Thank you for your kind and repetitive reminder. In fact, I had just begun reading the note you mentioned when I was replying to @PaigeMiller yesterday. I am now fully informed of the fact that weights should be multiplied when it comes to diagnosing collinearity in generalized linear models.

Still, I have some questions:

(1) I noticed that the var argument of PROC STANDARD standardizes all of the independent variables in the logistic model (li, temp and cell). Now that collinearity exists only between variable temp and the intercept, does all of the independent variables have to be standardized?

The link talks about a case where one of the predictor variables is highly correlated with the Intercept. It says: "The variation proportions associated with this large condition index suggest that TEMP is collinear with the intercept." and it also says "By rescaling the predictors, the collinearity with the intercept can be removed."

So standardizing the variables will remove the effect of high correlations if there are only large correlations with the intercept. If there are high correlations between one predictor variable (not the intercept) and another predictor variable (not the intercept), then standardizing the variables does not help remove this high correlation. (And so, the link is correct for the specific case of correlation with the intercept, it does not apply in general to high correlations between predictor variables).

Re: How can I perform principal component analysis for logistic regression via SAS?

StatDave — Mon, 27 Feb 2023 16:52:22 GMT

I don't think that standardization is a general solution to collinearity and its effect on the model.

I take another look at that cancer remission example in my note on penalty-based selection methods (LASSO, ridging, and elastic net). See the introductory text and, in particular, Example 3 which shows how the problem found in the first note on collinearity can be addressed using penalty-based selection. The LASSO method is shown using NLMIXED, which shows how the penalty is applied to the binomial likelihood when fitting the model, and with HPGENSELECT which simplifies its use. It also shows how the dual-penalty (LASSO+ridging), elastic net method can be applied using NLMIXED. The point is that collinearity, whether involving the intercept or other predictors, can be avoided using these shrinkage methods to select effects to stay in the model.

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Tue, 28 Feb 2023 15:21:52 GMT

Thank you, Paige, for your reply! Another issue I concern is the nomenclature of the statistics generated in the process we have been focused on in the past few days.

Another user of the community raised a question of the computation of GVIF in logistic regression model. I wonder if the VIF computed from the weighted information matrix can still be termed as "VIF". Should VIF calculated from weighted information matrix in generalized linear models be termed as "GVIF"?

Re: How can I perform principal component analysis for logistic regression via SAS?

Season — Tue, 28 Feb 2023 15:45:34 GMT

Thank you for your reply! With the help of you and @PaigeMiller , I think that I have a grasped the flowchart of dealing with collinearity in generalized linear models. Thank you both for your patience, kindness and expertise!

I have been insisting on consulting the problem of dealing with collinearity in the context of using the "ordinary" logistic regression model. The underlying reasons for this are as follows: (1) I hardly know anything about ridge regression and LASSO other than the names of the statistical methods and the most basic knowledge about them (they do not adopt unbiased estimation and that they are suitable for dealing with collinearity). (2) It has been reported that a larger sample is required for modern penalization methods like LASSO and elastic net when it comes to building clinical prediction models, but the author of the paper I cited simply stated that "further research on sample size requirement on methods like LASSO is required", thus possibly implying that no conclusion has been reached on that issue. In this case, sample size and power calculation may be a tough issue, at least for researchers building logistic regression as a clinical prediction model.