topic ERROR: Unable to allocate sufficient memory for the nonparametric correlation statistics. in Statistical Procedures

ERROR: Unable to allocate sufficient memory for the nonparametric correlation statistics.

aiste — Fri, 25 Sep 2020 09:29:30 GMT

Hello,

I am getting the following error 'ERROR: Unable to allocate sufficient memory for the nonparametric correlation statistics.' when I try to compute Spearman correlation coefficients. Anyone know how to overcome this problem?

Huge thanks in advance!

Re: ERROR: Unable to allocate sufficient memory for the nonparametric correlation statistics.

SteveDenham — Fri, 25 Sep 2020 11:13:14 GMT

First, check what memsize is set to in your SAS config file. Increasing the value there may solve the problem.

I suspect you have a pretty large dataset (multiple gigabytes?). One thing you might consider is using PROC MIANALYZE in a non-standard way.

First, use PROC SURVEYSELECT to subsample your data into multiple smaller datasets. Then use PROC CORR separately on each of these, adding the variable _imputation_ to each output. You could then run PROC MIANALYZE to get a model averaged result. See the example Combining Correlation Coefficients in the PROC MIANALYZE documentation for a way to get started..

SteveDenham

Re: ERROR: Unable to allocate sufficient memory for the nonparametric correlation statistics.

Rick_SAS — Fri, 25 Sep 2020 15:02:48 GMT

If your data set is really huge, I think Steve's suggestion can be simplified. You can simply use PROC SURVEYSELECT or the DATA step to take a random subset (a few million observations) and the use PROC CORR to get estimates from the sample.

Re: ERROR: Unable to allocate sufficient memory for the nonparametric correlation statistics.

SteveDenham — Mon, 28 Sep 2020 13:22:02 GMT

Not only is @Rick_SAS 's approach simpler, it will almost surely get around all of your memory problems. If a couple million records still overwhelms memory, then there are other issues that need to be addressed first.

SteveDenham