Statistical programming, matrix languages, and more

Proc IML Memory Limits

Reply
Occasional Contributor
Posts: 5

Proc IML Memory Limits

Hello,

I'm currently trying to run a hierarchical model on my dataset using a macro written by someone else that utilizes the Proc IML statement. However, I keep running into an error that I'm trying to figure out if it's my computer or SAS that is the problem.

The error I get is:

"Unable to allocate sufficient memory. At least 2147483647 more bytes required."

I've tried running it on my computer as well as a more powerful computer in our department (mine has 16GB ram, and the other has 32GB ram) and get the exact same error. I saw this post earlier in the year about limitations to the proc IML memory capacity and wanted to confirm that my understanding is that SAS is unable to work with larger datasets in the Proc IML statement (in 64-bit Windows 7) and that nothing can be done about it until, hopefully, SAS 9.4?

I have tried the memsize option and it doesn't make any difference. The code works since I tried using a small subset of my dataset to run the analyses and with about 5% of my observations, the macro works.

Thanks.

Respected Advisor
Posts: 2,655

Re: Proc IML Memory Limits

Andrew,

I think you are right about the memory thing.  What you might try is a jackknife approach, and resample your large dataset into smaller frames that will run, then combining the results.

Steve Denham

Occasional Contributor
Posts: 5

Re: Proc IML Memory Limits

Hi Steve,

Do you have any recommendations on how best to split up the data? In a nutshell, the main dataset has about 225,000 observations (rows = people) with a comparison to approximately 106 variables. The second stage HLM takes the 106 and tries to group them. I'm not that familiar with SAS's ability to run logistic regressions on subsets and how to combine them.

Thanks,

Andrew

SAS Super FREQ
Posts: 3,618

Re: Proc IML Memory Limits

A (225000 x 106) matrix isn't very large...only about 0.2 GB. You ought to have room for dozens of matrices of that size on your computer with 16GB of RAM.

Perhaps the macro uses many matrices of that size and does not free matrices that are no longer in use?  If so, you might be able to tweak the macro.

If you want to try Steve's suggestion, here is an article that describes how to read a portion of the data at a time. Perhaps you will find it useful.

Reading big data in the SAS/IML language - The DO Loop

Or, if the macro is too complicated to pull apart, you can try using the FIRSTOBS= and OBS= data set options to subset the data within a macro loop.

Ask a Question
Discussion stats
  • 3 replies
  • 731 views
  • 0 likes
  • 3 in conversation