Expected PROC OPTMODEL [convergence] differences between Base SAS 9.4 M5 and M6 ?

[Skeeze]

Hi all,

Not sure if you will be able to help, because I can't (easily) give a reproducible dataset/code currently. But I have a task that has been running in batch mode for 7ish months with no issues, however this month we did a run under M6 and got failure to converge. I reran on a multiple other M5 installs and found it converged fine - with the same numbers.

The optimisation problem is: creating a diversified risk margin measures in an Insurance context, which was previously done using Excel Goal Seek.

I've compared the outputs, and they are fairly close between the M5 (converging) and M6 (non-converging), but I'm worried that in the future it might not be. 

Does anyone have any insight around what might cause this?

EDIT: Digging into the logs it seems like under M6 it's stopping at 10 Iterations, under M5 it's 5000 iterations. Same code being run in batch,

Is there any syntax to force the iterations limit? I can't see this in the documentation.

Logs Under M5:

NOTE: Problem generation will use 4 threads.
NOTE: The problem has 3 variables (3 free, 0 fixed).
NOTE: The problem has 0 linear constraints (0 LE, 0 EQ, 0 GE, 0 range).
NOTE: The problem has 0 nonlinear constraints (0 LE, 0 EQ, 0 GE, 0 range).
NOTE: The OPTMODEL presolver removed 1 variables, 0 linear constraints, and 0 nonlinear constraints.
NOTE: The OPTMODEL presolved problem has 2 variables, 0 linear constraints, and 0 nonlinear constraints.
NOTE: Using analytic derivatives for objective.
NOTE: The NLP solver is called.
NOTE: The Interior Point algorithm is used.
                        Objective                          Optimality
           Iter             Value     Infeasibility             Error
              0        3897298400                 0        0.91352918
              1        3897061246                 0        0.91350405
              2        3896586953                 0        0.91345379
              3        3895638418                 0        0.91335329
              4        3893741557                 0        0.91315236
              5        3889948669                 0        0.91275076
              6        3882366228                 0        0.91194867
              7        3867214656                 0        0.91034889
              8        3836964540                 0        0.90716677
              9        3776674762                 0        0.90087152
             10        3656923924                 0        0.88855043
             11        3420636114                 0        0.86493619
             12        2960162758                 0        0.82145964
231 The SAS System                                                                                              11:28 Thursday, February 4, 2021

             13        2082352407                 0        0.74712617
             14         466179593                 0        0.63535466
             15         314803402                 0        0.59116425
             16          82986172                 0        0.61295189
             17          16022816                 0        0.60739250
             18           8757803                 0        0.60877532
             19           3625471                 0        0.60808359
             20           2567927                 0        0.60842937
             21   528332.02827454                 0        0.60825646
             22   245760.17213440                 0        0.60829968
             23   141279.05287933                 0        0.60827807
             24    52242.27852631                 0        0.60828888
             25    44517.95750427                 0        0.60828347
             26     3862.26792908                 0        0.60828617
             27     2185.25864410                 0        0.60828584
             28      838.50507355                 0        0.60828601
             29      673.37659454                 0        0.60828592
             30       82.56423950                 0        0.60828596
             31       11.92839050                 0        0.60828596
             32       11.69474030                 0        0.60828596
             33        0.11677551                 0        0.60828596
             34        0.06782532                 0        0.60828596
             35        0.02450562                 0        0.60828596
             36        0.02157593                 0        0.60828596
             37        0.00144196                 0        0.60828596
             38        0.00001526                 0        0.60828596
...
   4998        0.00001526                 0        0.60828596
           4999        0.00001526                 0        0.60828596
           5000        0.00001526                 0        0.60828596
NOTE: Maximum number of iterations reached.

Logs Under M6:

NOTE: Problem generation will use 4 threads.
NOTE: The problem has 3 variables (3 free, 0 fixed).
NOTE: The problem has 0 linear constraints (0 LE, 0 EQ, 0 GE, 0 range).
NOTE: The problem has 0 nonlinear constraints (0 LE, 0 EQ, 0 GE, 0 range).
NOTE: The OPTMODEL presolver removed 1 variables, 0 linear constraints, and 0 nonlinear constraints.
NOTE: The OPTMODEL presolved problem has 2 variables, 0 linear constraints, and 0 nonlinear constraints.
NOTE: Using analytic derivatives for objective.
NOTE: The NLP solver is called.
NOTE: The Interior Point Direct algorithm is used.
                        Objective                          Optimality
           Iter             Value     Infeasibility             Error
              0        3897298400                 0       14.43133695
              1        2368948288                 0       13.07313813
              2        1291680859                 0       26.10938827
              3          86140006                 0      435.11121099
              4           1009002                 0    37438.62334924
              5    51750.31443787                 0   730024.59935092
              6    20710.01023865                 0           1824203
              7     3789.87590790                 0           9968445
              8       75.42141724                 0         500907901
              9       14.59787750                 0        2587991583
             10        0.01017761                 0       37779184053
NOTE: Did not converge.
NOTE: Objective of the best feasible solution found = 0.0101776123.
NOTE: The best feasible solution found is returned.

[Skeeze]

Actually looking at it again, the Optimality error is behaving very differently under M6, it's growing and I guess as a result the iterations are stopping early.

[RobPratt · Accepted solution]

The MAXITER= option enforces a maximum number of iterations.  The default algorithm for the NLP solver changed from Interior Point to Interior Point Direct.  To force Interior Point, specify the solver option algorithm=INTERIORPOINT.

[Skeeze]

Thanks Rob, just tested it and I'm getting the same output it seems now. At least in the logs.

Where's the best place to read up on the differences here between DIRECT and Non-DIRECT? Or what drove the change in behaviour?

[RobPratt]

Glad to help.  The differences are highlighted in the documentation here.  The IPDIRECT algorithm generally performs best, which is why it is now the default.