BookmarkSubscribeRSS Feed
AndrewP
Calcite | Level 5

Hello,

I'm currently trying to run a hierarchical model on my dataset using a macro written by someone else that utilizes the Proc IML statement. However, I keep running into an error that I'm trying to figure out if it's my computer or SAS that is the problem.

The error I get is:

"Unable to allocate sufficient memory. At least 2147483647 more bytes required."

I've tried running it on my computer as well as a more powerful computer in our department (mine has 16GB ram, and the other has 32GB ram) and get the exact same error. I saw this post earlier in the year about limitations to the proc IML memory capacity and wanted to confirm that my understanding is that SAS is unable to work with larger datasets in the Proc IML statement (in 64-bit Windows 7) and that nothing can be done about it until, hopefully, SAS 9.4?

I have tried the memsize option and it doesn't make any difference. The code works since I tried using a small subset of my dataset to run the analyses and with about 5% of my observations, the macro works.

Thanks.

3 REPLIES 3
SteveDenham
Jade | Level 19

Andrew,

I think you are right about the memory thing.  What you might try is a jackknife approach, and resample your large dataset into smaller frames that will run, then combining the results.

Steve Denham

AndrewP
Calcite | Level 5

Hi Steve,

Do you have any recommendations on how best to split up the data? In a nutshell, the main dataset has about 225,000 observations (rows = people) with a comparison to approximately 106 variables. The second stage HLM takes the 106 and tries to group them. I'm not that familiar with SAS's ability to run logistic regressions on subsets and how to combine them.

Thanks,

Andrew

Rick_SAS
SAS Super FREQ

A (225000 x 106) matrix isn't very large...only about 0.2 GB. You ought to have room for dozens of matrices of that size on your computer with 16GB of RAM.

Perhaps the macro uses many matrices of that size and does not free matrices that are no longer in use?  If so, you might be able to tweak the macro.

If you want to try Steve's suggestion, here is an article that describes how to read a portion of the data at a time. Perhaps you will find it useful.

Reading big data in the SAS/IML language - The DO Loop

Or, if the macro is too complicated to pull apart, you can try using the FIRSTOBS= and OBS= data set options to subset the data within a macro loop.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

Multiple Linear Regression in SAS

Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.

Find more tutorials on the SAS Users YouTube channel.

From The DO Loop
Want more? Visit our blog for more articles like these.
Discussion stats
  • 3 replies
  • 1808 views
  • 0 likes
  • 3 in conversation