topic Re: Proc corr for a high number of variables, Please Help ..Thank You in SAS Procedures

Proc corr for a high number of variables, Please Help ..Thank You

Kanyange — Mon, 15 Dec 2014 15:04:24 GMT

Hi All,

I have around 1500 variables, I would like to check the correlation between variabes before I start using the variables for modelling. is there any smarter way to do it that doesn't take a long time to run, basically I wan't to remove anthing with a high correlation > 0.8 for example! Kind of macro but doesn't use do loop as I have millions of rows (8 Millioms) to process and our system is quite slow

Your help would be much appreciated..

Many Thanks

Re: Proc corr for a high number of variables, Please Help ..Thank You

PaigeMiller — Mon, 15 Dec 2014 15:14:08 GMT

is there any smarter way to do it that doesn't take a long time to run,

I'm going to say the answer is NO.

The reason I say this is that, in my opinion, looking at correlations is not the smartest way to proceed here. A better way to proceed, in my opinion, is to fit a PLS model to the data and determine combinations of your original variables that are predictive of the response. PLS takes into account the correlations between your variables better than most modelling techniques. Your method of looking at correlations between variables ignores whether or not the variable is predictive of the response.

Re: Proc corr for a high number of variables, Please Help ..Thank You

Kanyange — Mon, 15 Dec 2014 17:39:31 GMT

Thank you Paige...any documentation on PLS results interpretation?

Re: Proc corr for a high number of variables, Please Help ..Thank You

PaigeMiller — Mon, 15 Dec 2014 19:38:01 GMT

SAS Documentation for Proc PLS is a good place to start.

http://support.sas.com/documentation/cdl/en/statug/67523/HTML/default/viewer.htm#statug_pls_examples01.htm

Re: Proc corr for a high number of variables, Please Help ..Thank You

Kanyange — Mon, 15 Dec 2014 22:02:40 GMT

Thank you..so what do you think of Proc Reg to get the VIF?