BookmarkSubscribeRSS Feed
td1234
Calcite | Level 5

I have two questions.  I keep running into insufficient resources when using GLMSelect to investigate a file with 1M records, 5 categorical covariates with split options and 1 by variable.   If I replace the by statement with a macro, will this reduce the computational requirements and increase the probability the code will run?  Or are there just too many covariates with split options?

 

I can't increase the sumsize which is 1GB.  Any other suggestions?

 

3 REPLIES 3
Astounding
PROC Star

Switching to macro language will never reduce the resources needed.

 

If it is mathematically possible, you might lower the resource requirement by changing another variable, and turning it into a second BY variable.

PGStats
Opal | Level 21

If at all feasable, you could reduce the number of categories in some of your variables by merging similar categories.

 

If all else fails, you could run the model selection procedure on a subsample your data.

PG
SteveDenham
Jade | Level 19

Have you considered subsetting your data into, say, 10 subsets, each with approx. 100K records, using the MODELAVERAGE statement on each of these, and then comparing the results across the subsets (sort of a doubly averaged model) to get the selected variables, and then fitting the full dataset to the selected variables?  An adaptive LASSO method for variable selection would enable you to do this.  See Example 49.5 Model Averaging in the SAS/STAT14.1 documentation as a starting point.

 

Steve Denham

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 3 replies
  • 830 views
  • 1 like
  • 4 in conversation