Morning all,
I'm using Proc IML's NLPQN to solve a minimization problem with 9 nonlinear nonequality constraints. The choice variable is a vector with 16,000+ nonnegative entries. The problem is that the NLPQN call returns the " Unable to allocate sufficient memory" error. I've run structurally similar problems before with no issues, so I think the dimensionality is what's chocking the program here. I'm running SAS version 9.04.01M6P111518 in a machine with Nvidia 4090 GPU and an Intel Core i9-13900KF CPU with DDR5 RAM just shy of 60GB of RAM. My questions are: (i) is this a computationally feasible problem? (ii) I'm relatively new to SAS nonlinear programming, so can you give me some light on if and where the code can be more efficient? Thanks in advance, this community has a been a great resource since I started my SAS journey.
proc iml;
/* Store the status-quo param as a vector */
use work.param;
read all var {lparam} into pre_param0;
param0 = exp(pre_param0);
/* Storing the number of employees */
dim = nrow(param0);
/* Retrieve the coefficient columns as vectors */
use work.olsmat;
read all var {g1} into g1;
read all var {g2} into g2;
read all var {g3} into g3;
read all var {g4} into g4;
read all var {g5} into g5;
read all var {g2_g1} into g2_g1;
read all var {g4_g1} into g4_g1;
read all var {g3_g1} into g3_g1;
read all var {g5_g1} into g5_g1;
/* Retrieve status quo tots */
tot0 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0);
print tot0;
/* Declare initial guess */
z0 = j(1,dim,0);
/* Defining the objective function */
start rev(x);
S = sum(x);
return(S);
finish rev;
/* Declaring the gradient of the obj function */
start gradient(x) global(dim);
g = j(1,dim,1);
return(g);
finish gradient;
/* Defining lower and upper bounds */
noupper = j(dim,1,.);
lower = j(dim,1,0);
bounds_noupper = lower` // noupper`;
/* Defining the nonlinear constraints */
start constr(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
prov = log(param0`+x);
matc = j(9,1,.);
matc[1,1] = prov*g1;
matc[2,1] = prov*g2;
matc[3,1] = prov*g3;
matc[4,1] = prov*g4;
matc[5,1] = prov*g5;
matc[6,1] = prov*g2_g1;
matc[7,1] = prov*g3_g1;
matc[8,1] = prov*g4_g1;
matc[9,1] = prov*g5_g1;
return(matc);
finish constr;
/* Declaring the Jacobian of the constraint */
start jacobian(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
prov = param0`+x;
jac1 = g1`/prov;
jac2 = g2`/prov;
jac3 = g3`/prov;
jac4 = g4`/prov;
jac5 = g5`/prov;
jac6 = g2_g1`/prov;
jac7 = g4_g1`/prov;
jac8 = g3_g1`/prov;
jac9 = g5_g1`/prov;
return(jac1 // jac2 // jac3 // jac4 // jac5 // jac6 // jac7 // jac8 // jac9);
finish jacobian;
/* Defining Problem Options */
optn= j(1,11,.);
optn[2]=1; /* output printing options */
optn[10]=3; /* number of nonlinear constraints */
optn[11]=0; /* number of equality constraints */
/* Define Termination Criteria */
terc = j(1,13,.);
terc[1] = 10000;
terc[2] = 25000;
terc[4] = 1E-6;
terc[6] = 1E-6;
terc[8] = 1E-6;
/* Call the optimization method */
call nlpqn(rc, xres, "rev", z0, optn, bounds_noupper) tc=terc nlc="constr" grd="gradient" jacnlc="jacobian";
print rc;
optimum = xres`;
tot1 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0+optimum);
print tot1;
create optout from optimum[colname="rem"];
append from optimum;
close optout;
qui
t;
Please run this and share the result that will appear in the LOG:
proc options option=memsize;run;
For example on my system that shows:
MEMSIZE=2147483648 Specifies the limit on the amount of virtual memory that can be used during a SAS session.
which is the default 2G for a windows system.
If yours shows similar it may be possible to increase your MEMSIZE to use more memory. However that option is set at startup. You should share which operating system you are using to help us describe the easiest way to try an increased memory setting.
I get
SAS (r) Proprietary Software Release 9.4 TS1M6
MEMSIZE=62155645440
Specifies the limit on the amount of virtual memory that can be used during a SAS session.
OS is a Windows 10 Pro, and the processor is a 13th Gen Intel(R) Core(TM) i9-13900KF 3.00 GHz, 63.8 GB of usable RAM
That looks like SAS has all the 60G memory available, so the possibility of increasing memory doesn't look like a possible solution.
I'm not greatly familiar with IML so don't have any further suggestions.
I suspect @Rick_SAS will show up and may have some ideas or questions to clarify the situation.
And you can also add the following line
-memsize 80G
at the end of your SASV9.CFG to make your sas session have more memory.
From the program, it looks like you have 9 inequality constraints in the CONSTR function. But you have specified only three constraints in the option vector:
optn[10]=3; /* number of nonlinear constraints */
I suggest you try using 9 instead. If I'm misunderstanding, then please explain more about the problem you are trying to solve.
1. Can you post the SAS log that you get when you run the program?
2. You said, "The choice variable is a vector with 16,000+ nonnegative entries." Does that mean WORK.PARAM and WORK.OLSMAT datasets have 16,000+ values? If not, what is the number of rows for those datasets?
Instead of trying to attach it, can you paste it into the SAS Code Window by clicking the "Insert SAS Code" icon (aka, the "running man icon" )?
119
120 proc iml worksize=60155645440;
NOTE: Worksize = 262128
NOTE: Symbol size = 262128
NOTE: IML Ready
121
122 /* Store the status-quo param0 as a vector */
123 use work.param0;
124 read all var {psq} into pre_param0;
125 param0 = exp(pre_param0);
126
127 /* Storing the number of employees */
128 dim = nrow(param0);
129
130 /* Retrieve the coefficient columns as vectors */
131 use work.olsmat;
132 read all var {g1} into g1;
133 read all var {g2} into g2;
134 read all var {g3} into g3;
135 read all var {g4} into g4;
136 read all var {g5} into g5;
137 read all var {g2_g1} into g2_g1;
138 read all var {g4_g1} into g4_g1;
139 read all var {g3_g1} into g3_g1;
140 read all var {g5_g1} into g5_g1;
141
142 /* Retrieve status quo tots */
143 tot0 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0);
144 print tot0;
145
146 /* Declare initial guess */
147 z0 = j(1,dim,0);
148 r0 = j(1,dim,0.1);
149
150 /* Defining the objective function */
151 start rev(x);
152
152 ! S = sum(x);
153
153 ! return(S);
154 finish rev;
NOTE: Module REV defined.
5 The SAS System 11:56 Thursday, October 10, 2024
155
156 /* Declaring the gradient of the obj function */
157 start gradient(x) global(dim);
158
158 ! g = j(1,dim,1);
159
159 ! return(g);
160 finish gradient;
NOTE: Module GRADIENT defined.
161
162 /* Defining lower and upper bounds */
163 cap = 0.05;
164 upper = cap*param0;
165 noupper = j(dim,1,.);
166 lower = j(dim,1,0);
167 bounds = lower` // upper`;
168 bounds_noupper = lower` // noupper`;
169
170 /* Defining the nonlinear constraints */
171 start constr(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
172
172 ! prov = log(param0`+x);
173
173 ! matc = j(9,1,.);
174
174 ! matc[1,1] = prov*g1;
175
175 ! matc[2,1] = prov*g2;
176
176 ! matc[3,1] = prov*g3;
177
177 ! matc[4,1] = prov*g4;
178
178 ! matc[5,1] = prov*g5;
179
179 ! matc[6,1] = prov*g2_g1;
180
180 ! matc[7,1] = prov*g3_g1;
181
181 ! matc[8,1] = prov*g4_g1;
182
182 ! matc[9,1] = prov*g5_g1;
183
183 ! return(matc);
184 finish constr;
NOTE: Module CONSTR defined.
185
186 /* Declaring the Jacobian of the constraint */
187 start jacobian(x) global(param0, g1, g2, g3, g4, g5, g2_g1, g4_g1, g3_g1, g5_g1);
188
188 ! prov = param00`+x;
189
189 ! jac1 = g1`/prov;
190
190 ! jac2 = g2`/prov;
191
191 ! jac3 = g3`/prov;
192
6 The SAS System 11:56 Thursday, October 10, 2024
192 ! jac4 = g4`/prov;
193
193 ! jac5 = g5`/prov;
194
194 ! jac6 = g2_g1`/prov;
195
195 ! jac7 = g3_g1`/prov;
196
196 ! jac8 = g4_g1`/prov;
197
197 ! jac9 = g5_g1`/prov;
198
198 ! return(jac1 // jac2 // jac3 // jac4 // jac5 // jac6 // jac7 // jac8 // jac9);
199 finish jacobian;
NOTE: Module JACOBIAN defined.
200
201 /* Defining Problem Options */
202 optn= j(1,11,.);
203 optn[2]=1;
203 ! /* output printing options */
204 optn[10]=9;
204 ! /* number of nonlinear constraints */
205 optn[11]=0;
205 ! /* number of equality constraints */
206
207 /* Define Termination Criteria */
208 terc = j(1,13,.);
209 terc[1] = 10000;
210 terc[2] = 25000;
211 terc[4] = 1E-5;
212 terc[6] = 1E-5;
213 terc[8] = 1E-5;
214
215 /* Call the optimization method */
216 call nlpqn(rc, xres, "rev", z0, optn, bounds_noupper) tc=terc nlc="constr" grd="gradient" jacnlc="jacobian";
ERROR: (execution) Unable to allocate sufficient memory.
operation : NLPQN at line 216 column 1
operands : *LIT1065, z0, optn, bounds_noupper, terc, , , *LIT1067, *LIT1066, *LIT1068
*LIT1065 1 row 1 col (character, size 3)
rev
z0 1 row 16591 cols (numeric)
optn 1 row 11 cols (numeric)
bounds_noupper 2 rows 16591 cols (numeric)
terc 1 row 13 cols (numeric)
*LIT1067 1 row 1 col (character, size 8)
gradient
*LIT1066 1 row 1 col (character, size 6)
constr
*LIT1068 1 row 1 col (character, size 8)
7 The SAS System 11:56 Thursday, October 10, 2024
jacobian
statement : CALL at line 216 column 1
217
218 print rc;
ERROR: Matrix rc has not been set to a value.
statement : PRINT at line 218 column 1
219 optimum = xres`;
ERROR: (execution) Matrix has not been set to a value.
operation : ` at line 219 column 15
operands : xres
xres 0 row 0 col (type ?, size 0)
statement : ASSIGN at line 219 column 1
220
221 tot1 = (g1` // g2` // g3` // g4` // g5` // g2_g1` // g4_g1` // g3_g1` // g5_g1`)*log(param0+optimum);
ERROR: (execution) Matrix has not been set to a value.
operation : + at line 221 column 92
operands : param0, optimum
param0 16591 rows 1 col (numeric)
optimum 0 row 0 col (type ?, size 0)
statement : ASSIGN at line 221 column 1
222 print tot1;
ERROR: Matrix tot1 has not been set to a value.
statement : PRINT at line 222 column 1
223
224 create optout from optimum[colname="rem"];
ERROR: Matrix optimum has not been set to a value.
statement : CREATE at line 224 column 1
225 append from optimum;
ERROR: No data set is currently open for output.
statement : APPEND at line 225 column 1
226 close optout;
NOTE: Cannot close WORK.OPTOUT; it is not open.
227
228 quit;
NOTE: Exiting IML.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE IML used (Total process time):
real time 0.61 seconds
cpu time 0.57 seconds
I think the jacobian of the NLC is wrong. In the CONSTR function you define
matc[7,1] = prov*g3_g1;
matc[8,1] = prov*g4_g1;
but in the JACOBIAN function you define
jac7 = g4_g1`/prov; /* jac7 and jac8 need to be switched */
jac8 = g3_g1`/prov;
Are you running the program on a Windows machine?
Based on your description, I am going to guess that the issue is that your data are so big (16591 obs) that the Jacobian matrix exceeds 2GB. In fact, as discussed in the article, How much RAM do I need to store that matrix? - The DO Loop (sas.com)
the jacobian matrix for your problem occupies 2.05 GB. Some of the legacy computational routines have 2GB allocation limits, and I know that NLPQN first appeared in SAS version 6, so it classifies as a "legacy" procedure.
I suspect if you read less than 16,000 observations, the routine will not complain about out of memory. Of course, it could encounter other issues.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.
Find more tutorials on the SAS Users YouTube channel.