How do I use modules within Do loop groups

[Rick_SAS]

Just so you know, your program is doing 5000x5000 = 25 million MLEs of the Weibull distribution. during each iteration, the optimization routine must evaluate the objective function 6-8 times (however many iterations are required), plus another 24-32 times for numerical derivatives because you haven't supplied a derivative function.  Thus your objective function is being called about six BILLION times!

Here are a few suggestions to speed up your program.

1. Vectorize the objective function by getting rid of the loop over the data elements.
2. Provide a (vectorized) derivative function
3. In your DO loops there are many variables that are being defined over and over again. Pull those outside of the loops. For example: con, n, thet, optn, x2..
4. Make better guesses for the optimization.  Guess x=a||b instead of {0.5 0.5}.

If this still requires more time that you can afford, then try restructuring your program.  From what you've said, the simulation and MLE for the Weib(12.1, 4) samples are INDEPENDENT of the Weib(1,1) samples and MLEs.  To me, that implies that there really isn't a need to have two nested loops.  In one loop at the top of the program you can simulate, say, 10,000 Weib(1,1) samples and find the MLEs. Save those estimates in a 10000x2 array.  Then, use a single loop in which you generate 5000 Webull(12.1, 4) samples and compute the MLEs.  For each Weib(12.1, 4) sample, choose a RANDOM sample (with replacement) of  5000 parameter estimates from the precomputed array of 10000 parameter estimates.  (Use the SAMPLE function to choose 5000 estimates from the set 1:10000.) This should be functionally equivalent, but more than 1000 times faster because since you only need to compute15,000 MLEs instead of 25 million.

Good luck!

[IanWakeling]

I think you would be very wise to follow Rick's excellent suggestions.  Regarding the first suggestion, you could choose to make 'temp' a vector rather than a scalar, then there is no need for the do loop in your function module.

temp = g_x - thet;
sum1 = sum(log(temp));

You could calculate sum2 in a similar way.  In fact you could calculate sum1 and sum2 directly from g_x, since as far as I can see, the parameter 'thet' is always zero and can be ignored.

[timtimalo]

I appreciate your help and suggestions. I used the first four suggestions to speed up the program but I got the following error. (This program is first one in the attached file).

ERROR: (execution) Unable to allocate sufficient memory. At least 262128 more bytes required. How I can solve this problem?

 operation : NLPTR at line 77 column 1

 operands  : *LIT1059, X, OPTN, CON, , , , *LIT1060

*LIT1059     1 row       1 col     (character, size 7)

 f_weib2

X     1 row       2 cols    (numeric)

         1         1

OPTN     1 row       2 cols    (numeric)

         1         0

CON     2 rows      2 cols    (numeric)

      1E-6      1E-6

        .         .

*LIT1060     1 row       1 col     (character, size 7)

 g_weib2

 statement : CALL at line 77 column 1

When I used the rest of suggestions, the program run in two minutes and it resulted the same q (1-percent) for all different values of “a” for a specific sample size!

It also resulted the following error when “a=24.953” for sample size=15, 25, 50, 100 and in case that “a= 12.1543” for sample size 50.

ERROR: NEWRAP Optimization cannot be completed.

WARNING: Optimization routine cannot improve the function value.

I believe if there is something wrong in the program; it will not run and gives error for all “a” values and all sample sizes. So, why it gave the previous error in some cases?

How I can overcome this problem such that the program works correctly for all “a” values and all sample sizes (15, 25, 50, 100). (This program is the second one in the attached file).  

Thank you

[Rick_SAS]

Congratulations on reducing the run time so substantially! I'm glad the suggestions paid off.

I do not know the answer to your first question, but it doesn't matter since that first algorithm is inefficient and shouldn't be used.

The second issue (NEWRAP Optimization cannot be completed) is just saying that one of the 5000 Weibull(a,b) optimizations failed to converge.  Personally, I wouldn't worry about it.  It is probably the result of degenerate (simulated) data that doesn't look very much like a Weibull(a,b) distribution. If you are curious, you can try increasing the number of Newton iterations and see if the problem goes away.

However, given all the stochastic (random) things happening in this algorithm, you should be happy that 4999/5000=99.98% of your simulations provide useful results.  A sample size of 15 (or even 25) is very small for MLE estimation of the Weibull distribution. Some of your samples probably do not look very Weibull-y.

I don't understand your last two sentences that "something is wrong" and that the program "will not run and gives error for all “a” values and all sample sizes."  I am able to run your program to completion in my version of SAS 9.4.  It takes about 21 seconds in the SAS Windowing Environment (DMS) and in SAS Studio. Other than that one error message in the log, the program runs fine and produces a very nice distribution of widths (see attached).

[timtimalo]

Thank you for your help, I appreciate the efforts and the time you spent to review and answer my questions. Your suggestions were very helpful to complete the program which is an important part of my research.

In the last two sentences of my previous message I meant if there any modifications in the program that can help in getting rid of errors and warnings in case that simulating Weibull (a=24.953, b=4)?

When I simulated Weibull (“a=24.953” and “b=4”) for sample sizes 15, 25, 50 and 100, the number of errors and warnings increases by increasing the sample size. For example, Weibull (a=24.953, b=4) for “n=15”, I got the following errors and warnings:

ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value.

When I simulated Weibull (a=24.953, b=4) for “n=100”, I got the following errors and warnings: ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value. ERROR: NEWRAP Optimization cannot be completed. WARNING: Optimization routine cannot improve the function value.

I tried to increase the number of Newton iterations to 10000 but the problem does not go away.

In the previous cases, do simulations still provide useful results as you mentioned and there is no problem to use these results?

[Rick_SAS · Accepted solution]

I don't get that many errors, but as I said in my previous message, you can try to fiddle with the options vector for the NLP functions and see if that improves the situation.

I think you should mark this thread as "solved." If you have other questions later, open a new thread.