topic Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory in SAS/IML Software and Matrix Computations

NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Wed, 09 Oct 2024 18:57:15 GMT

Morning all,

I'm using Proc IML's NLPQN to solve a minimization problem with 9 nonlinear nonequality constraints. The choice variable is a vector with 16,000+ nonnegative entries. The problem is that the NLPQN call returns the " Unable to allocate sufficient memory" error. I've run structurally similar problems before with no issues, so I think the dimensionality is what's chocking the program here. I'm running SAS version 9.04.01M6P111518 in a machine with Nvidia 4090 GPU and an Intel Core i9-13900KF CPU with DDR5 RAM just shy of 60GB of RAM. My questions are: (i) is this a computationally feasible problem? (ii) I'm relatively new to SAS nonlinear programming, so can you give me some light on if and where the code can be more efficient? Thanks in advance, this community has a been a great resource since I started my SAS journey.

proc iml;

/* Store the status-quo param as a vector */
use work.param;
read all var {lparam} into pre_param0;
param0 = exp(pre_param0);

/* Storing the number of employees */
dim = nrow(param0);

/* Retrieve the coefficient columns as vectors */
use work.olsmat;
read all var {g1} into g1;                 
read all var {g2} into g2;             
read all var {g3} into g3;               
read all var {g4} into g4;           
read all var {g5} into g5;           
read all var {g2_g1} into g2_g1;      
read all var {g4_g1} into g4_g1;     
read all var {g3_g1} into g3_g1;       
read all var {g5_g1} into g5_g1;   

/* Retrieve status quo tots */
tot0 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0);
print tot0;

/* Declare initial guess */
z0 = j(1,dim,0);

/* Defining the objective function */
start rev(x);
	S = sum(x);
	return(S);
finish rev;

/* Declaring the gradient of the obj function */
start gradient(x) global(dim);
	g = j(1,dim,1);
	return(g);
finish gradient;

/* Defining lower and upper bounds */
noupper = j(dim,1,.);
lower = j(dim,1,0);
bounds_noupper = lower` // noupper`;

/* Defining the nonlinear constraints */
start constr(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
	prov = log(param0`+x);
	matc = j(9,1,.);
  	matc[1,1] = prov*g1;
	matc[2,1] = prov*g2;
	matc[3,1] = prov*g3;
	matc[4,1] = prov*g4;
	matc[5,1] = prov*g5;
	matc[6,1] = prov*g2_g1;
	matc[7,1] = prov*g3_g1;
	matc[8,1] = prov*g4_g1;
	matc[9,1] = prov*g5_g1;
	return(matc);
finish constr;

/* Declaring the Jacobian of the constraint */
start jacobian(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
	prov = param0`+x;
	jac1 = g1`/prov;
	jac2 = g2`/prov;
	jac3 = g3`/prov;
	jac4 = g4`/prov;
	jac5 = g5`/prov;
	jac6 = g2_g1`/prov;
	jac7 = g4_g1`/prov;
	jac8 = g3_g1`/prov;
	jac9 = g5_g1`/prov; 
	return(jac1 // jac2 // jac3 // jac4 // jac5 // jac6 // jac7 // jac8 // jac9);
finish jacobian;

/* Defining Problem Options */
optn= j(1,11,.); 
optn[2]=1;  /* output printing options */
optn[10]=3; /* number of nonlinear constraints */ 
optn[11]=0; /* number of equality constraints */

/* Define Termination Criteria */
terc = j(1,13,.);
terc[1] = 10000;
terc[2] = 25000;
terc[4] = 1E-6;
terc[6] = 1E-6;
terc[8] = 1E-6;

/* Call the optimization method */
call nlpqn(rc, xres, "rev", z0, optn, bounds_noupper) tc=terc nlc="constr" grd="gradient" jacnlc="jacobian";

print rc;
optimum = xres`;

tot1 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0+optimum);
print tot1;

create optout from optimum[colname="rem"];
append from optimum;
close optout;

quit;

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

ballardw — Wed, 09 Oct 2024 19:31:32 GMT

Please run this and share the result that will appear in the LOG:

proc options option=memsize;run;

For example on my system that shows:

 MEMSIZE=2147483648
                   Specifies the limit on the amount of virtual memory that can be used during a
                   SAS session.

which is the default 2G for a windows system.

If yours shows similar it may be possible to increase your MEMSIZE to use more memory. However that option is set at startup. You should share which operating system you are using to help us describe the easiest way to try an increased memory setting.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Wed, 09 Oct 2024 19:57:40 GMT

I get

    SAS (r) Proprietary Software Release 9.4  TS1M6

 MEMSIZE=62155645440
                   Specifies the limit on the amount of virtual memory that can be used during a SAS session.

OS is a Windows 10 Pro, and the processor is a 13th Gen Intel(R) Core(TM) i9-13900KF 3.00 GHz, 63.8 GB of usable RAM

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

ballardw — Wed, 09 Oct 2024 20:06:10 GMT

That looks like SAS has all the 60G memory available, so the possibility of increasing memory doesn't look like a possible solution.

I'm not greatly familiar with IML so don't have any further suggestions.

I suspect @Rick_SAS will show up and may have some ideas or questions to clarify the situation.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Wed, 09 Oct 2024 20:11:02 GMT

Sounds good, thanks for looking into it.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Ksharp — Thu, 10 Oct 2024 01:41:55 GMT

And you can also add the following line

-memsize 80G

at the end of your SASV9.CFG to make your sas session have more memory.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 13:21:19 GMT

From the program, it looks like you have 9 inequality constraints in the CONSTR function. But you have specified only three constraints in the option vector:
optn[10]=3; /* number of nonlinear constraints */

I suggest you try using 9 instead. If I'm misunderstanding, then please explain more about the problem you are trying to solve.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 14:11:20 GMT

1. Can you post the SAS log that you get when you run the program?

2. You said, "The choice variable is a vector with 16,000+ nonnegative entries." Does that mean WORK.PARAM and WORK.OLSMAT datasets have 16,000+ values? If not, what is the number of rows for those datasets?

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Thu, 10 Oct 2024 16:48:45 GMT

You're right, I changed it to optn[10]=9.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Thu, 10 Oct 2024 17:06:33 GMT

2. WORK.PARAM and WORK.OLSMAT have 16,000+ rows, as does my x. The datasets supply vectors that I use to build the constraint set. I'm looking for the x that minimizes rev while making all the constraint elements nonnegative.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Thu, 10 Oct 2024 17:19:03 GMT

It seems that my reply with the log is getting flagged as spam. I'll posting it again in a few minutes.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 17:19:18 GMT

I think the jacobian of the NLC is wrong. In the CONSTR function you define

	matc[7,1] = prov*g3_g1;
	matc[8,1] = prov*g4_g1;

but in the JACOBIAN function you define

	jac7 = g4_g1`/prov;   /* jac7 and jac8 need to be switched */
	jac8 = g3_g1`/prov;

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 17:20:58 GMT

Instead of trying to attach it, can you paste it into the SAS Code Window by clicking the "Insert SAS Code" icon (aka, the "running man icon" )?

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Thu, 10 Oct 2024 19:01:42 GMT

119        
120        proc iml worksize=60155645440;
NOTE: Worksize    = 262128
NOTE: Symbol size = 262128
NOTE: IML Ready
121        
122        /* Store the status-quo param0 as a vector */
123        use work.param0;
124        read all var {psq} into pre_param0;
125        param0 = exp(pre_param0);
126        
127        /* Storing the number of employees */
128        dim = nrow(param0);
129        
130        /* Retrieve the coefficient columns as vectors */
131        use work.olsmat;
132        read all var {g1} into g1;
133        read all var {g2} into g2;
134        read all var {g3} into g3;
135        read all var {g4} into g4;
136        read all var {g5} into g5;
137        read all var {g2_g1} into g2_g1;
138        read all var {g4_g1} into g4_g1;
139        read all var {g3_g1} into g3_g1;
140        read all var {g5_g1} into g5_g1;
141        
142        /* Retrieve status quo tots */
143        tot0 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0);
144        print tot0;
145        
146        /* Declare initial guess */
147        z0 = j(1,dim,0);
148        r0 = j(1,dim,0.1);
149        
150        /* Defining the objective function */
151        start rev(x);
152        	
152      !  S = sum(x);
153        	
153      !  return(S);
154        finish rev;
NOTE: Module REV defined.
5                                                          The SAS System                           11:56 Thursday, October 10, 2024

155        
156        /* Declaring the gradient of the obj function */
157        start gradient(x) global(dim);
158        	
158      !  g = j(1,dim,1);
159        	
159      !  return(g);
160        finish gradient;
NOTE: Module GRADIENT defined.
161        
162        /* Defining lower and upper bounds */
163        cap = 0.05;
164        upper = cap*param0;
165        noupper = j(dim,1,.);
166        lower = j(dim,1,0);
167        bounds = lower` // upper`;
168        bounds_noupper = lower` // noupper`;
169        
170        /* Defining the nonlinear constraints */
171        start constr(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
172        	
172      !  prov = log(param0`+x);
173        	
173      !  matc = j(9,1,.);
174          	
174      !    matc[1,1] = prov*g1;
175        	
175      !  matc[2,1] = prov*g2;
176        	
176      !  matc[3,1] = prov*g3;
177        	
177      !  matc[4,1] = prov*g4;
178        	
178      !  matc[5,1] = prov*g5;
179        	
179      !  matc[6,1] = prov*g2_g1;
180        	
180      !  matc[7,1] = prov*g3_g1;
181        	
181      !  matc[8,1] = prov*g4_g1;
182        	
182      !  matc[9,1] = prov*g5_g1;
183        	
183      !  return(matc);
184        finish constr;
NOTE: Module CONSTR defined.
185        
186        /* Declaring the Jacobian of the constraint */
187        start jacobian(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
188        	
188      !  prov = param00`+x;
189        	
189      !  jac1 = g1`/prov;
190        	
190      !  jac2 = g2`/prov;
191        	
191      !  jac3 = g3`/prov;
192        	
6                                                          The SAS System                           11:56 Thursday, October 10, 2024

192      !  jac4 = g4`/prov;
193        	
193      !  jac5 = g5`/prov;
194        	
194      !  jac6 = g2_g1`/prov;
195        	
195      !  jac7 = g3_g1`/prov;
196        	
196      !  jac8 = g4_g1`/prov;
197        	
197      !  jac9 = g5_g1`/prov;
198        	
198      !  return(jac1 // jac2 // jac3 // jac4 // jac5 // jac6 // jac7 // jac8 // jac9);
199        finish jacobian;
NOTE: Module JACOBIAN defined.
200        
201        /* Defining Problem Options */
202        optn= j(1,11,.);
203        optn[2]=1;
203      !             /* output printing options */
204        optn[10]=9;
204      !             /* number of nonlinear constraints */
205        optn[11]=0;
205      !             /* number of equality constraints */
206        
207        /* Define Termination Criteria */
208        terc = j(1,13,.);
209        terc[1] = 10000;
210        terc[2] = 25000;
211        terc[4] = 1E-5;
212        terc[6] = 1E-5;
213        terc[8] = 1E-5;
214        
215        /* Call the optimization method */
216        call nlpqn(rc, xres, "rev", z0, optn, bounds_noupper) tc=terc nlc="constr" grd="gradient" jacnlc="jacobian";
ERROR: (execution) Unable to allocate sufficient memory.

 operation : NLPQN at line 216 column 1
 operands  : *LIT1065, z0, optn, bounds_noupper, terc, , , *LIT1067, *LIT1066, *LIT1068

*LIT1065      1 row       1 col     (character, size 3)

 rev
z0      1 row   16591 cols    (numeric)
optn      1 row      11 cols    (numeric)
bounds_noupper      2 rows  16591 cols    (numeric)
terc      1 row      13 cols    (numeric)

*LIT1067      1 row       1 col     (character, size 8)

 gradient

*LIT1066      1 row       1 col     (character, size 6)

 constr

*LIT1068      1 row       1 col     (character, size 8)

7                                                          The SAS System                           11:56 Thursday, October 10, 2024

 jacobian

 statement : CALL at line 216 column 1
217        
218        print rc;
ERROR: Matrix rc has not been set to a value.

 statement : PRINT at line 218 column 1
219        optimum = xres`;
ERROR: (execution) Matrix has not been set to a value.

 operation : ` at line 219 column 15
 operands  : xres

xres      0 row       0 col     (type ?, size 0)


 statement : ASSIGN at line 219 column 1
220        
221        tot1 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0+optimum);
ERROR: (execution) Matrix has not been set to a value.

 operation : + at line 221 column 92
 operands  : param0, optimum
param0  16591 rows      1 col     (numeric)

optimum      0 row       0 col     (type ?, size 0)


 statement : ASSIGN at line 221 column 1
222        print tot1;
ERROR: Matrix tot1 has not been set to a value.

 statement : PRINT at line 222 column 1
223        
224        create optout from optimum[colname="rem"];
ERROR: Matrix optimum has not been set to a value.

 statement : CREATE at line 224 column 1
225        append from optimum;
ERROR: No data set is currently open for output.

 statement : APPEND at line 225 column 1
226        close optout;
NOTE: Cannot close WORK.OPTOUT; it is not open.
227        
228        quit;
NOTE: Exiting IML.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE IML used (Total process time):
      real time           0.61 seconds
      cpu time            0.57 seconds

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 20:05:04 GMT

Are you running the program on a Windows machine?

Based on your description, I am going to guess that the issue is that your data are so big (16591 obs) that the Jacobian matrix exceeds 2GB. In fact, as discussed in the article, How much RAM do I need to store that matrix? - The DO Loop (sas.com)
the jacobian matrix for your problem occupies 2.05 GB. Some of the legacy computational routines have 2GB allocation limits, and I know that NLPQN first appeared in SAS version 6, so it classifies as a "legacy" procedure.

I suspect if you read less than 16,000 observations, the routine will not complain about out of memory. Of course, it could encounter other issues.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Thu, 10 Oct 2024 20:22:28 GMT

The Jacobian is 'only' 9-by-16,000, which should amount to roughly 0.001152 GB

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 20:25:36 GMT

Ah! Thanks, I was confused. In that case, I don't know why you are getting an out-of-memory error.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

EconJAZZ — Thu, 10 Oct 2024 20:53:08 GMT

Alright, thanks for checking, @Rick_SAS. I'll post the solution when I get it.

Re: NLPQN - ERROR: (execution) Unable to allocate sufficient memory

Rick_SAS — Thu, 10 Oct 2024 21:04:31 GMT

It might be insightful to find out how large the data can be before the problem occurs. For example, instead of
READ ALL VAR ...
you could try

%let size = 1000;
use work.olsmat;
read next &size var {g1 g2 g3 g4 g5 g2_g1 g4_g1 g3_g1 g5_g1};
close;

etc

%let nobs = 1000;