topic Re: How to build a model within 11,651 variables? in SAS Procedures

How to build a model within 11,651 variables?

call_me_elaine — Wed, 17 Dec 2014 22:55:08 GMT

Hello everyone,

I have a .txt dataset which has 11,653 variables and 179 observations (see attachment). The variable "name" is just the id, I don't need it. The variable "score" is the responsable variable and the remaining 11,651 variables are the explanatory variables. I know those 11,651 variables are not all important, so I need to select the most significant ones to build and fit a model. After I import the .txt file into SAS, I tried stepwise regression in PROC HPREG, but SAS reported insufficient memory. I cannot change the MEMSIZE option (now is 2G) casue I am running the code on the server in my univerisity.

Here is my code:

proc hpreg data = P2222;

model score = P12050301--P60598281; *P12050301 is the first explanatory variable, P60598281 is the last explanatory variable;

selection method = stepwise(select=sl sle=0.25 sls=0.25 maxeffects=170);

run;

Do I have other methods to select the best model? I am totally a beginner. Thank you for your help!

Elaine

Re: How to build a model within 11,651 variables?

Reeza — Wed, 17 Dec 2014 23:18:15 GMT

You need to reduce your variables before you do a regression.

Ideally you'd know something about all your variables and then you could apply business knowledge as well as statistical techniques.

One way is to use forward selection (generally not recommended) which is to regress each variable against the dependent and only use those that are significant. You also need to check for correlation between your independent variables.

You'll also need to consider the variable types, i.e. categorical, numerical, ordinal and treat them appropriately in your variable selection method.

Because you have 179 variables you'll want AT LEAST LESS THAT that in your regression, otherwise you have a dimensionality problem and your matrices won't invert.

Re: How to build a model within 11,651 variables?

PGStats — Wed, 17 Dec 2014 23:29:59 GMT

I don't know the solution to the performance issue here, but...

If your explanatory variables (P12050301--P60598281) were independent random numbers, unrelated to score, you would certainly find a model that fits your data perfectly. Fitting that model is almost like trying to solve N linear equations for N variables.

Try setting apart a small set of, say, 20 observations. Build your model based on the rest of the data and then test the resulting model on the small set.

Re: How to build a model within 11,651 variables?

stat_sas — Thu, 18 Dec 2014 02:03:01 GMT

Try data reduction techniques to identify variables explaining most of the variation before proceeding to modeling.

Re: How to build a model within 11,651 variables?

call_me_elaine — Thu, 18 Dec 2014 19:46:35 GMT

Hi, thank you for your reply. I understand what you mean here. All my variables are numerical. I don't know something about those variables so I cannot select them using prior knowledge. That's why I use stepwise regression and let SAS select the important ones. But right now because of the memory probelm, I cannot get the result.

Re: How to build a model within 11,651 variables?

call_me_elaine — Thu, 18 Dec 2014 19:58:19 GMT

Hi, thank you for your reply. I may not quite understand you. I tried the stepwise regression but SAS reported insufficient memory. This means the explanatoy variables are too many (11,651), so SAS don't have enough memory for this proc.

Did you mean I separate the dataset into several small ones and build a model on each of the small dataset? Actually I tried it. I sepatated the whole dataset into 2 small ones, the first one has the first 5,826 explanatory variables and the second one has the remaining 5,825 variables. Then I run the stepwise regression on both of them and built 2 models. The first model selected 10 variables and R square=0.24, the second model selected 155 variables and R square=1.0. But if I separte the whole dataset into different way, say the first 5,000 variables and then 6,651 variables, those model selected different variables and R square is different.

Now I am really confusing about it. How can I separate the dataset and get the "best" result?

Thanks,

Elaine

Re: How to build a model within 11,651 variables?

call_me_elaine — Thu, 18 Dec 2014 19:59:39 GMT

You mean reduction methods like PCA? I used stepwise regression, don't SAS will select the important variables automatically?

Thanks,

Elaine

Re: How to build a model within 11,651 variables?

stat_sas — Thu, 18 Dec 2014 21:11:49 GMT

Yes, Factor analysis/PCA will provide few dimensions that can be used in stepwise regression.

Re: How to build a model within 11,651 variables?

PGStats — Thu, 18 Dec 2014 21:16:42 GMT

Hi Elaine,

Divide your dataset the other way by creating two classes of observations (not variables) with a weight variable:

data splitData;

set myData;

if rand("UNIFORM") < (21/179) then weight = 0;

else weight = 1;

run;

proc reg data=splitData;

model score = ... ;

weight weight;

output out=outData p=predScore;

run;

proc corr data=outData(where=(weight=0)) Pearson;

var score; with predScore;

run;

proc sgplot data=outData(where=(weight=0));

scatter x=score y=predScore;

run;

"the second model selected 155 variables and R square=1.0". This is what I warned you about. Contrary to intuition, you have very little chance of creating a meaningful model from that many variables via a variable selection method. The biggest challenge you face is your lack of knowledge about these variables. They can probably be grouped into closely related subsets that could each be represented by one best variable or a single Principal Component.

Good luck!

Re: How to build a model within 11,651 variables?

stat_sas — Thu, 18 Dec 2014 21:34:01 GMT

Hi PG,

This is an interesting way. Just to learn more about your recommended method, could you please provide more info on this, like how are getting 155 variables?

Regards,

Naeem

Re: How to build a model within 11,651 variables?

SteveDenham — Fri, 19 Dec 2014 18:53:00 GMT

I like thinking about this one in the following way: The OP measured over 11K variables about 180 times, so we have nearly 60 times as many variables as cases. You can RANDOMLY select any 179 of the variables and get a PERFECT fit to the response variable. In fact there are (thank you Wolfram Alpha) :

176958671964634377740198691965577713682480110474545351945677139031026937863665740570836480756356258632972678963277

474969010136808169597920994461712218986603493340022182644089536811284759493681290314266912794597881035605378960409

924200421757503946290949030028695642411309948914571542814736217126680552692397498583913574843384467106219631759603

05286963384521740723958639266557563907788742389725738156000 possible perfect fits (that's 1.78E400), and stepwise regression will pick one of those. Given that the total number of protons and electrons in the universe is on the order of 300 orders of magnitude less than this, you may consider stepwise methods as an exercise in futility. Finding the "right" one is impossible in P Log P time.

So the best I could think of is to include the response variable into the mix and look at principal components. Find those that have large loadings on the dependent variable, and explain most of the variability.

Actually, the best idea is to find a subject matter expert and whittle the 11K plus variables down to, say, 8 or 10, which is about how many your 179 cases can accurately estimate.

Steve Denham

Re: How to build a model within 11,651 variables?

PaigeMiller — Fri, 19 Dec 2014 19:09:38 GMT

Partial Least Squares Regression was designed for this case. It is often used when the number of variables far exceeds the number of data points, for example, in spectroscopy, where you might have measured the intensity at 10,000 wavelengths on 179 samples. You can find lots of examples in the literature of PLS models for spectroscopy that were similar to your case with many times more X variables than observations.

With PLS, you get data reduction in the sense that it will find linear combinations of your X variables that are predictive of Y. Better than PCA or factor analysis, where you get lienar combinations of your X variables that might just be non-predictive since the Y values are not used in determining the PCA/factor analysis dimensions.

But PLS won't give you individual X variables that are in the model, it's not designed to, and as other have pointed out, there is no logical way to pick individual Xs from your 11,651 Xs that are in the model. So give up this idea of using stepwise regression.

Re: How to build a model within 11,651 variables?

SteveDenham — Fri, 19 Dec 2014 19:16:28 GMT

True! And it can incorporate 's work regarding validation, using the CROSSVAL options.

I was still afraid that 11K right-hand side variables could overwhelm even PLS, but given your endorsement for that size, I will defer. It is the method for use.

And if the OP is interested in variable reduction, they can use the output as an exploratory tool in identifying strongly determining factors and strongly redundant variables.

Steve Denham

Re: How to build a model within 11,651 variables?

PaigeMiller — Fri, 19 Dec 2014 19:27:15 GMT

Well, I didn't say the OP has enough hardware resources, but that's a different issue. PLS doesn't require inverting a matrix if you use the NIPALS Algorithm (which is the default in PROC PLS), so it doesn't really require huge amounts of memory, and its pretty fast. If her machine can handle it, then that's the way to go.

Re: How to build a model within 11,651 variables?

call_me_elaine — Fri, 19 Dec 2014 21:01:25 GMT

I did try PCA and used the first 160 components. The MSE is 0.0106. But I didn't get the coefficient of these components and what are these components. I got the results but just don't know how to explain it.

Re: How to build a model within 11,651 variables?

call_me_elaine — Fri, 19 Dec 2014 22:09:15 GMT

Hi PG,

Thanks again for your reply. Yes you are right. There are so many variables and I don't have enough knowledge about them. So simply using regression didn't give me good results.

I tried decision tree and PCA today. Decison tree selected 15 variables out of 11,651 and MSE is 0.139. For PCA method, there are 160 components in the model and MSE is 0.0106. However, I didn't get the coefficient of these components, so I don't know how to explain it.

You talked about dividing the original dataset into 2 small ones by separating the observations. One is used for building a model, the other is used for testing. Why can't just use all the observations to build the model and then test it? Will that make the result more accurate and reasonable? And what is weight meaning and what its purpose?

Best wishes!

Elaine

Re: How to build a model within 11,651 variables?

call_me_elaine — Fri, 19 Dec 2014 23:15:58 GMT

Thanks so much for your reply. You just gave me so much information and I need some time to understand it since I am totally totally a new user. At least I know stepwise is not the solution.

BTW, what is OP short for?

Re: How to build a model within 11,651 variables?

call_me_elaine — Fri, 19 Dec 2014 23:17:12 GMT

Thanks so much for your reply. You just gave me so much information and I need some time to understand it since I am totally totally a new user. At least I know stepwise is not the solution.:)

Re: How to build a model within 11,651 variables?

art297 — Fri, 19 Dec 2014 23:22:45 GMT

OP is short for Original Poster.

I'm not a statistician, but have taken numerous statistic's courses. I think that factor analysis, or PCP, will be a good first step to reduce your number of variables. I.e., you could create combined scores that take into account collections of grouped measures. However, that would require business knowledge to determine whether the results (i.e., what to combine) seem to make sense.

PG's proposal was simply to create a model on a sample of your data and see if it held up on another sample of your data.

Re: How to build a model within 11,651 variables?

call_me_elaine — Fri, 19 Dec 2014 23:36:04 GMT

Yes that's what I am thinking now. Do the variable reduction first. I tried PCA today and there are 160 components in the model where MSE is 0.0106. However, I didn't get the coefficient of these components, so I don't know how to explain it.

I guess what PG means is the N fold cross-validation.

Thanks,

Elaine