Proc IML Memory Limits

[AndrewP]

Hello,

I'm currently trying to run a hierarchical model on my dataset using a macro written by someone else that utilizes the Proc IML statement. However, I keep running into an error that I'm trying to figure out if it's my computer or SAS that is the problem.

The error I get is:

"Unable to allocate sufficient memory. At least 2147483647 more bytes required."

I've tried running it on my computer as well as a more powerful computer in our department (mine has 16GB ram, and the other has 32GB ram) and get the exact same error. I saw this post earlier in the year about limitations to the proc IML memory capacity and wanted to confirm that my understanding is that SAS is unable to work with larger datasets in the Proc IML statement (in 64-bit Windows 7) and that nothing can be done about it until, hopefully, SAS 9.4?

I have tried the memsize option and it doesn't make any difference. The code works since I tried using a small subset of my dataset to run the analyses and with about 5% of my observations, the macro works.

Thanks.

[SteveDenham]

Andrew,

I think you are right about the memory thing.  What you might try is a jackknife approach, and resample your large dataset into smaller frames that will run, then combining the results.

Steve Denham

[AndrewP]

Hi Steve,

Do you have any recommendations on how best to split up the data? In a nutshell, the main dataset has about 225,000 observations (rows = people) with a comparison to approximately 106 variables. The second stage HLM takes the 106 and tries to group them. I'm not that familiar with SAS's ability to run logistic regressions on subsets and how to combine them.

Thanks,

Andrew

[Rick_SAS]

A (225000 x 106) matrix isn't very large...only about 0.2 GB. You ought to have room for dozens of matrices of that size on your computer with 16GB of RAM.

Perhaps the macro uses many matrices of that size and does not free matrices that are no longer in use?  If so, you might be able to tweak the macro.

If you want to try Steve's suggestion, here is an article that describes how to read a portion of the data at a time. Perhaps you will find it useful.

Reading big data in the SAS/IML language - The DO Loop

Or, if the macro is too complicated to pull apart, you can try using the FIRSTOBS= and OBS= data set options to subset the data within a macro loop.