Cannot Get the Co-efficients to match up - Optimizing the error

[david27]

Team & @RobPratt 

Please look into the below code.

First I am assigning values to w1-w8 and b1-b3.

Then i am verifying that everything is populated in the correct fashion in the verification_dsn dataset.

I am then using proc optmodel to validate if i reach similar/same numbers for w1-w8 and b1-b3.

But everything comes out as 0.

I had this example in excel and then using solver, i reached the numbers for w1-w8 and b1-b3.

The formulas for minimize function and the constraint are exactly same.

For example: If a+b=c then c-b=a. And I am just trying to validate that.

But something is a miss here. 
Please advise.

Thank You.

@mkeintz: Tagging you because I thought you must be interested in the solution.

data equation;
input y m x c;
datalines;
14 9 1 5
851 9 94 5
473 9 52 5
635 9 70 5
518 9 57 5
257 9 28 5
383 9 42 5
428 9 47 5
356 9 39 5
68 9 7 5
257 9 28 5
248 9 27 5
194 9 21 5
86 9 9 5
761 9 84 5
284 9 31 5
653 9 72 5
356 9 39 5
320 9 35 5
;run;
%let w1=0.094961;
%let w2=0.008734;
%let w3=6.21478;
%let w4=7.002409;
%let w5=0.012339;
%let w6=0.396723;
%let w7=13.94726;
%let w8=1.045836;

%let b1=0;
%let b2=0;
%let b3=0;
data verification_dsn;
	set equation;
w1=0.094961;
w2=0.008734;
w3=6.21478;
w4=7.002409;
w5=0.012339;
w6=0.396723;
w7=13.94726;
w8=1.045836;

b1=0;
b2=0;
b3=0;
	n1h1=x*&w1.-&b1.;
	n2h1=x*&w2.-&b1.;
	n1h2=((x*&w1.-&b1.)*&w3.)+((x*&w2.-&b1.)*&w4.)-&b2.;
	n2h2=(n1h1*&w5.)+(n2h1*&w6.)-&b2.;
	output_network=(n1h2*&w7.)+(n2h2*&w8.)-&b3.;
	sq_diff=(y-output_network)**2;
	minimize_function=(y-(((((x*w1-b1)*w3)+((x*w2-b1)*w4)-b2)*w7)+((((x*w1-b1)*w5)+((x*w2-b1)*w6)-b2)*w8)-b3))**2;
	constraint=abs(((((x*w1-b1)*w3)+((x*w2-b1)*w4)-b2)*w7)+((((x*w1-b1)*w5)+((x*w2-b1)*w6)-b2)*w8)-b3);
run;

proc optmodel;
	var w1 ,w2 ,w3 ,w4 ,w5 ,w6 ,w7 ,w8 ,b1 ,b2 ,b3;
	number x ,y;
	read data equation into x y;
	min sq_diff=(y-(((((x*w1-b1)*w3)+((x*w2-b1)*w4)-b2)*w7)+((((x*w1-b1)*w5)+((x*w2-b1)*w6)-b2)*w8)-b3))**2;
	con y = abs(((((x*w1-b1)*w3)+((x*w2-b1)*w4)-b2)*w7)+((((x*w1-b1)*w5)+((x*w2-b1)*w6)-b2)*w8)-b3);
	solve;
	print w1 w2 w3 w4 w5 w6 w7 w8 b1 b2 b3;
	expand / iis;
quit;

[david27]

@RobPratt @mkeintz 

I think, it is just reading the first observation of the dataset and solving the first row only.

I see in the log---

NOTE: There were 1 observations read from the data set WORK.EQUATION.

This is not intended.

I want optmodel to read in the entire dataset and then come up with the parameters.

[RobPratt · Accepted solution]

As indicated in the log, you were reading only one observation:

NOTE: There were 1 observations read from the data set WORK.EQUATION.

If you add a PRINT statement after the READ DATA, you can see that only the first observation was read:

print x y;

x	y
1	14

Instead, you need an index set for the observations:

set OBS;
number x {OBS}, y {OBS};
read data equation into OBS=[_N_] x y;
print x y;

[1]	x	y
1	1	14
2	94	851
3	52	473
4	70	635
5	57	518
6	28	257
7	42	383
8	47	428
9	39	356
10	7	68
11	28	257
12	27	248
13	21	194
14	9	86
15	84	761
16	31	284
17	72	653
18	39	356
19	35	320

Similarly, the objective function needs to sum over the observations:

min sq_diff = sum{i in OBS}(y[i]-(((((x[i]*w[1]-b[1])*w[3])+((x[i]*w[2]-b[1])*w[4])-b[2])*w[7])+((((x[i]*w[1]-b[1])*w[5])+((x[i]*w[2]-b[1])*w[6])-b[2])*w[8])-b[3]))**2;

From your solution, it looks like you wanted w and b to be >= 0.  Here's a compact way to declare that:

var w {1..8} >= 0, b {1..3} >= 0;

By default, decision variables have infinite lower and upper bounds.  If you do not specify >= 0, it turns out the solver actually finds an optimal objective value of 0 for this problem.

I'm not sure what you intended with EXPAND / IIS because you are not using the IIS solver option.  But EXPAND by itself will show you the full problem as a sanity check.

The problem appears to be nonconvex, so I recommend using the MULTISTART (or MS) option to try to avoid getting stuck in a local minimum.

Putting it all together:

proc optmodel;
   var w {1..8} >= 0, b {1..3} >= 0;
   set OBS;
   number x {OBS}, y {OBS};
   read data equation into OBS=[_N_] x y;
   print x y;
   min sq_diff = sum{i in OBS}(y[i]-(((((x[i]*w[1]-b[1])*w[3])+((x[i]*w[2]-b[1])*w[4])-b[2])*w[7])+((((x[i]*w[1]-b[1])*w[5])+((x[i]*w[2]-b[1])*w[6])-b[2])*w[8])-b[3]))**2;
   expand;
   solve with nlp / ms;
   print w b;
quit;

The SAS System

The OPTMODEL Procedure

Problem Summary
Objective Sense	Minimization
Objective Function	sq_diff
Objective Type	Nonlinear
Number of Variables	11
Bounded Above	0
Bounded Below	11
Bounded Below and Above	0
Free	0
Fixed	0
Number of Constraints	0

---

The SAS System

The OPTMODEL Procedure

Solution Summary
Solver	Multistart NLP
Algorithm	Interior Point Direct
Objective Function	sq_diff
Solution Status	Optimal
Objective Value	126.64605979
Number of Starts	100
Number of Sample Points	1600
Number of Distinct Optima	30
Random Seed Used	1679670
Optimality Error	3.1111693E-8
Infeasibility	0
Presolve Time	0.00
Solution Time	0.55

[1]	w	b
1	43.23150674	-.0000000021
2	34.96859624	0.0000017686
3	102.26490433	-.0000000082
4	80.22402775
5	180.63897324
6	97.11066371
7	0.00055746
8	0.00045163

The resulting solution is different than what you expected, but the optimal objective value matches, so this looks like a case of multiple optimal solutions.

[david27]

@RobPratt :: You THE Boss.

Thank You very much for that.

I read this link numerous times but was not able to comprehend the part that you showed.

https://support.sas.com/documentation/onlinedoc/or/132/optmodel.pdf

What you showed here in the example-- where is that in this documentation of OPTMODEL?

Regards.

[RobPratt]

Glad to help.  Which part do you mean?

[david27]

Reading the entire dataset and then go to SOLVE part.

[RobPratt]

The READ DATA statement is documented here and is also illustrated in most of the end-of-chapter examples.  You might also be interested in this book of 29 examples, almost all of which use the READ DATA statement.

[david27]

Hello @RobPratt :

In the above example i did put a constraint:

con y = abs(((((x*w1-b1)*w3)+((x*w2-b1)*w4)-b2)*w7)+((((x*w1-b1)*w5)+((x*w2-b1)*w6)-b2)*w8)-b3);

Is the way I put the constraint correct or it also has to be array?

con y[i]=(((((x[i]*w[1]-b[1])*w[3])+((x[i]*w[2]-b[1])*w[4])-b[2])*w[7])+((((x[i]*w[1]-b[1])*w[5])+((x[i]*w[2]-b[1])*w[6])-b[2])*w[8])-b[3]);

I will have to add a constraint here where y for each observation is equal to value coming out of formula.

Because, if i don't do that, then Solver calculates a mean value that satisfies the solver condition and as a result the value is same for all rows.

[RobPratt]

Correct syntax is:

con c {i in OBS}: y[i] = ((((x[i]*w[1]-b[1])*w[3])+((x[i]*w[2]-b[1])*w[4])-b[2])*w[7])+((((x[i]*w[1]-b[1])*w[5])+((x[i]*w[2]-b[1])*w[6])-b[2])*w[8])-b[3];

But a best practice is to use an implicit variable if the same long expression appears more than once:

impvar yhat {i in OBS} = ((((x[i]*w[1]-b[1])*w[3])+((x[i]*w[2]-b[1])*w[4])-b[2])*w[7])+((((x[i]*w[1]-b[1])*w[5])+((x[i]*w[2]-b[1])*w[6])-b[2])*w[8])-b[3];
min sq_diff = sum{i in OBS}(y[i]-yhat[i])**2;
con c {i in OBS}: y[i] = yhat[i];