Out of memory when using proc model to estimate parameters

[Koala123]

Hi,

I’m trying to estimate some parameters, but I have some issues with the memory being too small. I’m estimating parameters using proc model. I have about 100 parameters and I use 10 000+ equations to estimate the parameters.

I run SAS 9.4 (64-bit Native Mode) on a PC with 16 GB RAM. I get the error message: “WARNING: Cannot get enough memory for task. Error flag is set; the task will not be performed.”

For testing purposes, I decrease the number of equations to about 9000. Let’s call this test program a. The code is working.

I do another test where I add about 100 equations to test program a. Let’s call this test program b. The code is not working.

I set MEMSIZE to maximum which increases the MEMSIZE from 2 GB to 7.5 GB. I tried running the test programs again. Test program a is working. Test program b is not working.

I also tried increasing MEMBLKSZ and  MEMMAXSZ. I get the same results: Test program a is working. Test program b is not working.

I then tried running the test programs using SAS 9.4 (64-bit native mode) on a PC with 32 GB RAM. Same result: Test program a is running. Test program b is not running. Increasing MEMSIZE to maximum (30 GB) does not help. Increasing MEMBLKSZ and  MEMMAXSZ to 6 GB does not help.

How do I find what kind of memory there is not enough of and how do I increase it?

[JosvanderVelden]

Can you provide the proc model syntax you are using so we can get a clearer picture of the tests and options used?

Are you using the latest SAS release or an earlier one. I believe the current production version is SAS 9.4 Maintenance Release 8 (SAS 9.4 TS1M8). Is your dataset large or relatively small?

[Koala123]

16 GB RAM laptop: SAS (r) Proprietary Software 9.4 (TS1M5 DBCS3170) (Installed a long time ago.)

32 GB RAM desktop: I don't have access to it right now, but I installed it today from university source, so I would think that is up to date.

I construct a test-dataset with about 1 250 000 observations.

Part of the code:

proc model data=ardata plots=none details;
array s2tr[50] s2tr_1968-s2tr_2017;
array s2pr[50] s2pr_1968-s2pr_2017;
endogenous var covar1-covar25;
exogenous year agecell;
parameters rho 0.9 s2etainit 0.1 s2tr_1968-s2tr_2017 0.04 s2pr_1968-s2pr_2017 0.015;
control s2merr=&s2merr;

y = year - 1968 + 1;

*setting up parameters;
eta2 = rho**(2*(agecell-24-1))*s2etainit;
do i1=2 to agecell-24;
i2 = y - (i1-2);
if i2<1 then i2=1;
eta2 = eta2 + rho**(2*(i1-2))*s2pr[i2];
end;

*euqations to be fitted;
if y=1 then do;
eq.var_y1968 = -var + eta2 + s2tr[y] + s2merr;
eq.covar_y1968_lag1 = -covar1 + rho*eta2;
eq.covar_y1968_lag2 = -covar2 + rho**2*eta2;
eq.covar_y1968_lag3 = -covar3 + rho**3*eta2;
eq.covar_y1968_lag4 = -covar4 + rho**4*eta2;
... (more equations) ...
eq.covar_y1968_lag22 = -covar22 + rho**22*eta2;
eq.covar_y1968_lag23 = -covar23 + rho**23*eta2;
eq.covar_y1968_lag24 = -covar24 + rho**24*eta2;
eq.covar_y1968_lag25 = -covar25 + rho**25*eta2;
end;

else if y=2 then do;
eq.var_y1969 = -var + eta2 + s2tr[y] + s2merr;
eq.covar_y1969_lag1 = -covar1 + rho*eta2;
eq.covar_y1969_lag2 = -covar2 + rho**2*eta2;
... (more equations) ...
eq.covar_y1969_lag24 = -covar24 + rho**24*eta2;
eq.covar_y1969_lag25 = -covar25 + rho**25*eta2;
end;

...

Many more years

...

fit var_y1968 covar_y1968_lag1-covar_y1968_lag25
var_y1969 covar_y1969_lag1-covar_y1969_lag25
var_y1970 covar_y1970_lag1-covar_y1970_lag25
var_y1971 covar_y1971_lag1-covar_y1971_lag25
var_y1972 covar_y1972_lag1-covar_y1972_lag25
var_y1973 covar_y1973_lag1-covar_y1973_lag24
var_y1974 covar_y1974_lag1-covar_y1974_lag23 covar_y1974_lag25
var_y1975 covar_y1975_lag1-covar_y1975_lag22 covar_y1975_lag24
var_y1976 covar_y1976_lag1-covar_y1976_lag21 covar_y1976_lag23 covar_y1976_lag25
var_y1977 covar_y1977_lag1-covar_y1977_lag20 covar_y1977_lag22 covar_y1977_lag24
var_y1978 covar_y1978_lag1-covar_y1978_lag19 covar_y1978_lag21 covar_y1978_lag23 covar_y1978_lag25
var_y1979 covar_y1979_lag1-covar_y1979_lag18 covar_y1979_lag20 covar_y1979_lag22 covar_y1979_lag24
var_y1980 covar_y1980_lag1-covar_y1980_lag17 covar_y1980_lag19 covar_y1980_lag21 covar_y1980_lag23 covar_y1980_lag25
var_y1981 covar_y1981_lag1-covar_y1981_lag16 covar_y1981_lag18 covar_y1981_lag20 covar_y1981_lag22 covar_y1981_lag24
var_y1982 covar_y1982_lag1-covar_y1982_lag15 covar_y1982_lag17 covar_y1982_lag19 covar_y1982_lag21 covar_y1982_lag23 covar_y1982_lag25

/ missing=pairwise converge=0.00001 maxiter=10000 outest=stock_params1 memoryuse;
bounds s2etainit s2tr_1968-s2tr_2017 s2pr_1968-s2pr_2017 >= 0;

run;

When I use equations from 1968-1982 the code works. I need to estimate parameters up until 2017 which means I have to use equations up until 2017. For the last part of the dataset I only have data every other year, so there are fewer equations towards the end of the dataset.

Thank you for taking the time!

[karemelsqary]

https://learn.sas.com/pluginfile.php/8878/mod_scormddl/content/28763/LWSPAGS901.pdf

[ChrisNZ]

Check what value you have for option memsize, which sets the total (real+paging) memory your session can use.

This option is set at startup, so only your SAS administrator can change it.

Other useful options to query are:

realmemsize to see how much real memory the SAS session is allowed to use. This option is also set at startup.

xmrlmem to see how much real memory is left at any point in time. This option cannot be set, only queried.

You can query them using function getoption().

[Season]

The getoption function seems a useful function. Could you please provide examples on how to use them to check the memory available? I checked SAS Help and found it seems not available as a statement that acts independently like "ODS GRAPHICS ON;". It seems that it must be embedded in an argument like 

if getoption('yearcutoff') = '1926' then
     do;

So if I am just launching a new SAS session and want to check out how much memory do I have, how should I use the getoption function to do this?

[ChrisNZ]

You need learn how to use SAS.

You can run something like

data _null_;
 A=getoption('nameoftheoption') ;
 putlog A=;
run;
    

[Season]

Thank you!

[Patrick]

@Koala123 

Just to make sure that your changes to memory settings are also active I'd suggest you run below code prior to the code that creates the out of memory condition.

proc options group=memory;
run;

[Koala123]

Hi, I run this code and the memory does increase. Thanks!

[SASKiwi]

If you add the FULLSTIMER option to the start of your program, your SAS log will report on the memory usage of each DATA or PROC step. You can then see how much you require versus how much you have:

options fullstimer;