MLE using a data step within PROC IML

[rakeshallu]

Dear all, 

Apologies for the multiple posts. I wanted to share the progress I was able to make with the previous question. In case it helps anyone help me.

I used 'submit' and 'endsubmit' to convert all the vector operations in my previous post into data steps. This enabled me to get rid of the Do-loop. However, with the code below, nlpnra is simply returning the initial value.  Any suggestions on what am I doing wrong will be very helpful. 

Thank you very much in advance. 

proc iml;
start LogLik(param);
p1 = param[1]; 
p2 = param[2]; 
p3 = param[3]; 
p4 = param[4]; 
p5 = param[5]; 

submit p1 p2 p3 p4 p5;

data _null_; 
	set want;
	/*scroll further probability*/
	if position < 3 then sc_p = 1; 
	else sc_p = logistic(&p1 + &p2*position + &p3*pred_usage + &p4*relevant_offer_cnt_wt + &p5*credits_r);  

	/*end probability*/ 
	if position < max_pos then ed = 0; 
	else ed = 1 - sc_p; 
	
	by seq; 
	retain prod_choice; 
	retain prod_scroll;
	retain ll; 
	retain total_ll 0;

	if first.seq then do; 
	prod_choice = 1; 
	prod_scroll = 1; end; 

	prod_choice = prod_choice*not_choosing_prob; 
	prod_scroll = prod_scroll*sc_p; 
	prod_choice_end = prod_choice*ed;

	lag_prod_scroll = lag(prod_scroll); 
	if first.seq then lag_prod_scroll = 1; 

	pos_end_ll = lag_prod_scroll*prod_choice_end; 

	if first.seq then ll = pos_end_ll; 
	else ll = ll + pos_end_ll; 
	if last.seq then do; 
	log_ll = log(ll);
	total_ll = total_ll + log_ll; 
	end;
call symput ("total_ll",total_ll);  
run; 
endsubmit;
C = &total_ll.;
return sum(C); 
finish;
param = {-0.1 -0.005 0.1 -0.1 0.2 0.0};
optn = {1, /* 3. find max of function, and */ 4}; /* print moderate amount of output */
con = {. . . . . ., 
	    . . . . . .};
call nlpnra(rc, xres, "LogLik", param, optn, con);
quit; 

Best, 

Rakesh

[PaigeMiller]

You can't have a data step within PROC IML.

[rakeshallu]

Thank you for the response.
We can do a data step within proc iml..
See Rick's video below -
https://blogs.sas.com/content/iml/2011/10/24/video-calling-sas-procedures-from-the-sasiml-language.h...

Get Outlook for Android<>

[Tom]

Are you sure this line is getting the value of the macro variable generated by the submitted data step?

C = &total_ll.;

Or does it just get the value of the macro variable that was present when the IML step started?

Does IML have something similar to the SYMGET() or SYMGETN() function of the data step?

[Rick_SAS]

> Does IML have something similar to the SYMGET() or SYMGETN() function of the data step?

You can call almost every DATA step function from an IML program. See Calling Base SAS Functions from SAS/IML Programs - The DO Loop

Specifically, you can certainly call SYMGET.

Both SYMGET and directly referencing the macro variable will work. For example, run this program:

proc iml;

SUBMIT;
data _null_;
  call symputx("test", 3.14);
run;
ENDSUBMIT;

x = &test;
y = symget("test");
print x y;
quit;

 However, if you are using a DO loop, then use SYMGET, because the loop is parsed and compiled, which means that the macro resolution occurs at parse time, not run time. See Macros and loops in the SAS/IML language - The DO Loop

[rakeshallu]

Thank you, Rick and Tom. 

I see that symget works in the ml environment. However, in the line highlighted by Tom "C = &total_ll.;", the macro variable is not getting updated with each iteration of nlpnra. It simply retains the value before the start of proc iml. 

Any suggestions on how do I update this variable with nlpnra? 

[Rick_SAS · Accepted solution]

> Apologies for the multiple posts. 

@rakeshallu I was going to look at your issue, but now I don't know which one is active.

Regarding your attempt to use the SUBMIT block inside an objective function to a nonlinear optimization, I would discourage this approach. It will be harder to develop, debug, and understand this hybrid approach. Furthermore, the objective function needs to be fast because it will be called potentially hundreds of times during the optimization. There is considerable overhead costs to using the SUBMIT block, so I suspect that the performance will be better by using a native SAS IML function module as the objective function.

[rakeshallu]

Thank you, Rick. In light of this suggestion, we could put the submit/endsubmit thread to rest. 

In the previous post, I use the native SAS IML module to compute the objective function. Any suggestions on improving it would be very helpful.