Hi everybody,   I am optimizing an integer (binary) linear model with proc optmilp.  After many iterations, the solver runs out of memory.   The process was moving in the right direction started to plateau with a large gap (>300%) so I need to find ways to optimize this further.   Here is what I have done so far:  Changed the formulation of the model couple of times in order to use less variables. Aggregated my data and split the problem into multiple smaller problems (cover smaller geographic areas one by one instead of trying to solve one nationwide model.) Developed a method to give the optimization model a decent starting point (an initial feasible solution, so that it does not start from zeros.)  I am generating this initial solution with a clustering algorithm. Increased memory allocation (from 4 GB to 8, then 12, and recently up to 16 GB of memory.) I confirmed that the amount of memory requested is indeed being assigned to the SAS program (the logs show that.)   Confirmed that SAS has plenty of temp space. Have 400 GB available at the beginning of the process (and records that show that this space is not being consumed while the model is running.) Tweaked some solver options (set cutting planes to aggresive.)   This is not a complex or uncommon model (the heart of it is a simple assignment problem) so probably many ways exist to improve upon this.  I thought that I would share this with the SAS optimization community as people with more experience in this particular area may have lots to suggest.  I want to solve this with SAS… that would be my goal.   I created a “toy problem” with simulated test data (to protect the confidential details) but the structure of the model is essentially the same.  I am posting 3 SAS programs with their respective logs: optim_test_scenario1.sas: Small model. This one solves easily. No problems here. I created it just to test the process. The first part of the script creates the test dataset. The last part has the actual optimization model. optim_test_scenario2.sas: Medium model. Same model structure but larger input datasets (which translate into more variables and constraints.) This model ran out of memory (with 16 GB) but exited gracefully so I have a solution saved.  The solution was moving in the right direction (compared to the seed) but was not optimal.    optim_test_scenario3.sas: Large model. Same model structure but many more variables and constraints. This model ran out of memory (with 16 GB) and did not exit gracefully so I don’t have a solution saved.    My goal would be to make scenario2 work.  That would probably be enough.  But if there is a way to solve for something of the magnitude of scenario 3, that would be even better.   Here are the main questions in my mind: What ways may exist to formulate this problem more efficiently? using less variables, less resources, but solving for the same minimization goal (without oversimplifying the problem) What solver parameters or options may help this particular problem to converge more quickly (this is where many opportunities may exist, that would be my guess, I have not explored this enough, I see lots of possibilities and don’t have a lot of time to test/try them all one by one.) How much memory is typically needed for a problem of this size? (maybe I’m asking for too little) What do people typically use? Or how does one determine the right sizing?  (the log shows the size of the problem in terms of # variables, constraints, etc.) Is there a way to have SAS start using temp space (disk) when memory is running out?  So that the process does not abort.   Thanks to anybody that takes a look at this. I would appreciate any comments or feedback.     Carlos ... View more

Hi,   Is there a way to configure proc optmodel to gracefully exit (and save the solutions that it has found so far) when it runs out of memory?   I am solving a binary linear model with the MILP solver.  The relaxed problem finishes quickly but the branch and cut algorythm is getting stuck. After many iterations (1.5 days runing) the algorythm runs out of memory and terminates without saving the 7 solutions that it had found up to that point.  The gap was not small (~35%) so I may not want to use any of those solutions anyway, but I would like to see the best one, at the very least.  Unfortunatelty, the solver is terminating with status "OUT_OF_MEMORY", solution status "OUTMEM_NOSOL", and the create data statements saves zero (0) observations.   Here are some questions that come to mind in this situation: Is there an option for me to tell optmodel to save the best integer solution found (despite this exit status)? Or what could I do to have proc optmodel save a solution every "x" iterations or every "x" hours? e.g. gracefully exit solver with a timeout (I could use maxtime for that), save the best integer solution, and then continue solving from that same point (without having to start over again from the root of the search tree.)  I could include this in a loop, so that not everything is lost if we suddenly hit a "out of memory" error... but this only makes sense if there is a way to continue with the branching and cutting seamlesly.  Or is there a way to tell OPTMODEL to save the search tree in disk so that it never runs out of memory?  FYI I am using the system options to increase memory and cpu (-memsize 4096M -cpucount 8.)  According to the log, SAS is getting this amount of memory and the Branch and Cut algorithm is taking advantage of all 8 processors.  It seems to me that, if there was a way for optmodel to use disk when the tree becomes very large, this amount of memory could be enough... and solving the problem would be just a matter of waiting longer, solving just to 90%, etc. Of course, I am also tweaking the solver options (probe, nodesel, cutting plane levels, etc.) to see if there is a way to converge faster to the solution. So far, all the runs are ending up in a similar way. I am attaching the SAS log produced by the SOLVE statement in one of those runs.  It would be nice to be able to see what are the options producing (hence this question.)  Since the question is not specific to this particular problem, I am not including the model code itself, but let me know if any other information would help.   Would appreciate any pointers! ... View more