Thank you for your reply! First of all, I would like to answer my own question that was raised a few days ago (so that other users of the Community can save their time on looking up information): diagnostics of collinearity should precede the variable selection process. I looked up a book on multivariate statistics and referred to the part on collinearity (of multivariate linear regression), it said (original text not in English): "Multicollinearity (an alias of collinearity) is the distortion of the model estimation or inability of accurate estimation caused by the precise correlation of or the fact that strong correlation exists among the covariates (can be interpreted as "independent variables" in this setting) in the linear regression model. Therefore, prior to regression, knowing the relationship among the covariates is of great importance". The text I translated clear pointed out that diagnostics of collinearity should precede the variable selection process. I also consulted one of my teachers responsible for teaching us SAS. She also stated that diagnostics of collinearity should precede the variable selection process. As for the reason underlying my failure to diminish collinearity, I myself searched for an answer after raising my question. You mentioned that an inappropriate setting of dummy variables may be one of the underlying causes of the problem. I am gratitude for your pointing out that issue (so I will never make such a mistake in my upcoming data analysis), but unfortunately this was not the case in my problem. I read the note you had mentioned again and noticed that the underlying cause of collinearity between the independent variable and the intercept is the disproportionally small standard deviation of the variable that exhibit collinearity with the intercept. I reviewed my model and found out that I put one continuous independent variable alongside dummy variables in the model (the reason why I did so was that compared with other variables, the range of the continuous variable is relatively small, so in order to retain more information of my data, I put the continuous variable directly in the model) and that the mean of other independent variables (dummy variables) are close to 0.5, with their standard deviation ranging from 0.5-0.6. However, the continuous variable had a mean of around 6 and a standard deviation of around 1. The standard deviation of the continuous variable was smaller than its mean while the standard deviations of the discrete (dummy) variables were larger than their means. In other words, compared with the other independent variables, the standard deviation of the continuous variable was disproportionally small. In the first place, I tackled the problem by standardizing all the variables into the variables with 1 as their standard deviation (just like the case the note you referred). However, the largest condition index computed from the weighted information matrix was 11 prior to variable standardization (the second largest was 8, so there is no need to concern about that); the largest condition index computed from the weighted information matrix was 12 after variable standardization, with the very same variable still exhibiting collinearity with the intercept and that no collinearity was observed when intercept was removed from analysis. In other words, using 1 as the standard deviation in PROC STANDARD did not help to reduce collinearity at all.  I decided to try several different standard deviations in PROC STANDARD. First, I tried 0.5, producing even more severe collinearity (the largest condition index computed from the weighted information matrix was 29). Then, I tried 3. This time, collinearity disappeared. So, in conclusion, choosing the right standard deviation in PROC STANDARD is of vital importance in dealing with collinearity with the variable standardization method. One should not choose the standard deviation in PROC STANDARD arbitrarily, i.e. without observing the exact number of the statistics. ... View more

Hello, Paige. I tried to use the note provided by @StatDave to analyze my data today and encountered some problems. (1) Should diagnostics and treatment of collinearity precede or follow the variable selection process? That issue is what I had not think of prior to analyzing my own data. A dilemma of the researcher when collinearity is found is whether or not to trust the statistical results based on the analysis prior to diagnostics of collinearity. Explicitly speaking, collinearity can cause wrong results, thus leading to "fake" statistics (e.g. P-value of the independent variables, estimation of the regression coefficients (β)). In this case, the researcher cannot ascertain whether the insignificances of the independent variables are caused by collinearity or truth (it is insignificant per se), arbitrarily excluding the independent variables via model selection methods may result in the situation where truly significant variables be excluded from collinearity diagnostics and the final model building process. (2) I tried with my own data today and found out that my data perfected matched with the circumstance @StatDave demonstrated in the note he/she cited. Collinearities of four independent variables with the intercept were found, while no collinearity was found among the independent variables. Collinearity was not found if the intercept was excluded via the collinnoint argument in PROC REG. However, collinearity persisted after variable standardization. The very four variables that are proved to collinear with the intercept were still collinear with the intercept after all of the independent variables were standardized. What is more, collinearity still disappeared after excluding the intercept, indicating the four variables were still collinear with the intercept, even after variable standardization. What should I do now? Thank you for your suggestion! ... View more

Hello, I encountered problems when I was dealing with collinearity in logistic regression models with your note today. (1) Should diagnostics and treatment of collinearity precede or follow the variable selection process? That issue is what I had not think of prior to analyzing my own data. A dilemma of the researcher when collinearity is found is whether or not to trust the statistical results based on the analysis prior to diagnostics of collinearity. Explicitly speaking, collinearity can cause wrong results, thus leading to  "fake" statistics (e.g. P-value of the independent variables, estimation of the regression coefficients (β)). In this case, the researcher cannot ascertain whether the insignificances of the independent variables are caused by collinearity or truth (it is insignificant per se), arbitrarily excluding the independent variables via model selection methods may result in the situation where truly significant variables be excluded from collinearity diagnostics and the final model building process. (2) I tried with my own data today and found out that my data perfected matched with the circumstance you demonstrated in your note. Collinearities of four independent variables with the intercept were found, while no collinearity was found among the independent variables. Collinearity was not found if the intercept was excluded via the collinnoint argument in PROC REG. However, collinearity persisted after variable standardization. The very four variables that are proved to collinear with the intercept were still collinear with the intercept after all of the independent variables were standardized. What is more, collinearity still disappeared after excluding the intercept, indicating the four variables were still collinear with the intercept, even after variable standardization. What should I do now? ... View more

I have another question on the note you provided. Should VIF computed with the weighted information matrix still be called "VIF"; or "GVIF", as another user of SAS Community had mentioned?  Thank you! ... View more

Thank you for your reply! With the help of you and @PaigeMiller , I think that I have a grasped the flowchart of dealing with collinearity in generalized linear models. Thank you both for your patience, kindness and expertise! I have been insisting on consulting the problem of dealing with collinearity in the context of using the "ordinary" logistic regression model. The underlying reasons for this are as follows: (1) I hardly know anything about ridge regression and LASSO other than the names of the statistical methods and the most basic knowledge about them (they do not adopt unbiased estimation and that they are suitable for dealing with collinearity). (2) It has been reported that a larger sample is required for modern penalization methods like LASSO and elastic net when it comes to building clinical prediction models, but the author of the paper I cited simply stated that "further research on sample size requirement on methods like LASSO is required", thus possibly implying that no conclusion has been reached on that issue. In this case, sample size and power calculation may be a tough issue, at least for researchers building logistic regression as a clinical prediction model. ... View more

Thank you, Paige, for your reply! Another issue I concern is the nomenclature of the statistics generated in the process we have been focused on in the past few days. Another user of the community raised a question of the computation of GVIF in logistic regression model. I wonder if the VIF computed from the weighted information matrix can still be termed as "VIF". Should VIF calculated from weighted information matrix in generalized linear models be termed as "GVIF"? ... View more

They are not the same. By definition, odds ratio is the ratio of odds of those who are exposed and those who are not exposed to a certain exposure. It is most frequently used in case-control studies, where (in most cases) no follow-up occurs. Relative risk is the ratio of incidence of a certain event (usually called "endpoint") between those who are exposed and those who are not exposed to a certain exposure. Relative risk is used in cohort studies and experimental studies (e.g. randomized clinical trails, RCTs), where follow-up from the "zero time" (the time you start observing if the endpoint happened in the cohort) is a must. Odds ratio and relative risk are fundamental and basic concepts in epidemiology. You can refer to a book on epidemiology for further explanation on the two terms. ... View more

Oh, yes, you were totally correct (laugh oh laugh), I surely did not see yesterday. @PaigeMiller wrote: @Season wrote: OK, I see. Computing VIF in PROC REG when the dependent variable is a continuous one is easy. Yet the question I raised earlier is the computation of VIF in a logistic regression model. Can SAS do that? Thanks! No you don't see. I said: "To use the VIF in PROC REG, you create a made up variable that is a continuous Y and use your X-variables. The VIF does not depend on the Y variable." I did not completely understand what you mean yesterday and was merely focusing one the word "continuous" before the letter "Y". I thought you must have misunderstood me, since I made a lengthy reply yesterday, with my questions hidden between the lines. As a result, you may had just skimmed through my reply, without noticing that I was modeling a discrete independent variable. So that's the underlying reason for my further reply you quoted. And of course, there's also the point made by @StatDave  Now I think that I have truly seen what you meant. But other questions emerged when I was reading the note @StatDave cited. I have already raised my questions in my latest reply to @StatDave. I wonder if you could offer your suggestion to the three questions, if you don't mind. Thank you very much for your patience and your time spent on my questions again! ... View more

Thank you for your kind and repetitive reminder. In fact, I had just begun reading the note you mentioned when I was replying to @PaigeMiller yesterday. I am now fully informed of the fact that weights should be multiplied when it comes to diagnosing collinearity in generalized linear models. Still, I have some questions: (1) I noticed that the var argument of PROC STANDARD standardizes all of the independent variables in the logistic model (li, temp and cell). Now that collinearity exists only between variable temp and the intercept, does all of the independent variables have to be standardized? (2) The means of obliterating (or at least reducing) collinearity in a logistic regression model demonstrated here is variable standardization. In a complete model building process, what follows the PROC STANDARD procedure is using these standardized variables to perform logistic regression modeling. Eventually, the user may wish to transform the standardized variables into unstandardized ones. When I was a student studying statistics, my teacher demonstrated an example of using SAS to perform principal component analysis for multivariate linear regression. She completed the final process (i.e. transform the standardized variables back to the unstandardized ones after the entire model building process) by writing down the equation in hand and perform arithmetic calculations on her own. Is there an automatic way of doing that final transformation process by SAS? (3) The circumstance illustrated in the note you provided was one where one independent variable collinears with the intercept. What if the independent variables collinear with each other? Aside from deviating from the original model (i.e. switching to penalty-based model selection process like LASSO or other methods like Logistic Partial Least Squares Regression, etc.) and simply deleting one or more variables involved in collinearity, is variable standardization still a solution to that problem? If so, should the researcher standardize all the independent variables, as is the case in the note you provided; or just the independent variables that are involved in collinearity? Many thanks! ... View more

OK, I see. Computing VIF in PROC REG when the dependent variable is a continuous one is easy. Yet the question I raised earlier is the computation of VIF in a logistic regression model. Can SAS do that? Thanks! ... View more

Hi, there. I am also building a logistic regression model and is also troubled by collinearity. I consulted @PaigeMiller and @StatDave_SAS for the diagnostics and tackling of collinearity in logistic regression. In short, the conclusions are: (1) As @Ksharp has mentioned, SAS does not support computing GVIF. (2) Diagnostics of collinearities can be done by using principal component analysis (PROC PRINCOMP). But caution should be taken when it comes to lowering the dimension of the data using principal component analysis, since (some of) the dimensions it finds may not be good predictors of the dependent variable. (3) Therefore, it is recommended that the issue of collinearity causing ill-conditioning of the information matrix used in the model-fitting process (instead of lowering the dimension of the data) be handled by methods described in this note. (4) When it comes to lowering the dimension of the data, both Logistic Partial Least Squares Regression (not available in SAS, but available as a package in R) and penalty-based model selection process such as LASSO (available in PROC HPGENSELECT) can be good choices, with the latter (e.g. LASSO) selecting a subset of the candidate predictors rather than combining them all into a small number of functions. In this case, you may not bother to diagnose collinearity. You can take a look at my post (How can I perform principal component analysis for logistic regression via SAS?) to see the original replies by @PaigeMiller and @StatDave_SAS. Good luck! ... View more

OK, thank you very much for your help! I will read the note you have mentioned carefully and try LASSO as well to compare the two methods. It's too bad that SAS Community only supports accepting merely one reply as the solution. I think that your replies and the replies given by @PaigeMiller are all very fruitful for not only me, but also all of those that are troubled by collinearity in logistic regression. After all, I have retrieved nearly zero article discussing the solution of the collinearity problem in my search for articles on the Internet. Instead of discussing much about mathematical or statistical theories prior to providing a solution (like most articles do), your replies get straight to the point-- provide answers to the problem directly. I myself deem your replies as wonderful "concise textbooks" to the problem. I am sure that your replies can benefit other researchers who are struggling to find a solution to that problem and spending much time on searching for information instead of data analysis itself. By the way, I major in medicine and is familiar with a few search engines that specialize in searching for articles on medicine (e.g. PubMed). Could you please introduce the search engine statisticians frequently use (aside from Google Scholar) or a few prestigious journals on statistics? Thank you both for your kind help again! ... View more

Thank you very, very much, Koen, for your wonderful and helpful reply! I will investigate the information you provided in depth.  ... View more

Oh, by the way, I have another problem concerning the diagnostics (discovery) of collinearities in logistic regression. In linear regression models, tolerance, variance inflation factor (VIF), as well as condition index (computed from eigenvalues) can serve as indicators of collinearities among the independent variables in the model. The aforementioned three statistics can be computed in PROC REG upon request. However, they are not available in the modules that build logistic regression models (i.e. PROC LOGISTIC, PROC GENMOD, PROC HPLOGISTIC, etc.). Therefore, diagnostics of collinearity in logistic regression is not that easy. I tried PROC PRINCOMP in my data today and found out that PROC PRINCOMP does not compute the three statistics either. Instead, it produces a correlation matrix of the variables I wish to analyze. There is no surprise that "strong" correlations exist among the variables I put in the logistic regression model, with some of the correlation coefficient reaching 0.6154. I guess that collinearities must exist in this situation. So here are my questions: when it comes to diagnostics of collinearity, can correlation coefficients serve as surrogate statistics for tolerance, VIF and condition index in logistic regression? If not, what statistic(s) can do this job? Also, how can I compute tolerance, VIF and condition index in logistic regression? Could @PaigeMiller,  @StatDave or someone else kindly give me a hand? Thank you all very much! ... View more

Thank you for the help you have offered! Currently, my most important concern is the diagnostics of collinearity. I will take a look at LASSO and the note you have mentioned. ... View more