topic Re: Optimize the algorithm by using IML in SAS Programming

Optimize the algorithm by using IML

Jack2012 — Wed, 14 Oct 2020 05:46:08 GMT

My Dears,

With enlightenment from paper https://support.sas.com/resources/papers/proceedings/proceedings/forum2007/188-2007.pdf， I am worndering if there is a way to make the code running more efficient (in terms of the time it used to get the dataset), like convert to IML, using array?

%macro SIMON(usern, p0, p1, alpha, beta );
data stage1;
do n1=2 to &usern-1;
do r1=0 to n1 while (r1<n1);
term1_p0 = cdf('BINOMIAL', r1, &p0, n1);
term1_p1 = cdf('BINOMIAL', r1, &p1, n1);
if term1_p1=<&beta then output; /*remove solution sets that do not meet the beta requirement*/
end;
end;
run;

data stage12;
set stage1;
do n=n1+1 to &usern;
do r=r1+1 to n while (r1<r<n);
term2_p0=0; *initialize the summation terms for alpha & beta calculations;
term2_p1=0;
do x=r1+1 to min(r, n1);
dum0=pdf('BINOMIAL', x, &p0, n1)*cdf('BINOMIAL', r-x, &p0, n-n1);
dum1=pdf('BINOMIAL', x, &p1, n1)*cdf('BINOMIAL', r-x, &p1, n-n1);
term2_p0= term2_p0+dum0;*recursive formulae used for summation terms;
term2_p1= term2_p1+dum1;

if 1-(term1_p0+term2_p0)=<&alpha and (term1_p1+term2_p1)=<&beta then output; end;
end;
end;
run;
%mend;
%SIMON(alpha=0.05, beta=0.2, p0=0.05, p1=0.25, usern=130)/*Though the paper recommended not exceed 120, there is still the case that to check if there is more optimal one beyond 120*/

IML is strange to me, therefore I seek your help to optimze the above code. Thank you in advance.

Re: Optimize the algorithm by using IML

Jack2012 — Wed, 14 Oct 2020 05:47:48 GMT

If I am correct, loop in PROC SQL is not easy or maybe impossible.

Re: Optimize the algorithm by using IML

Ksharp — Wed, 14 Oct 2020 10:33:36 GMT

Calling @Rick_SAS

Re: Optimize the algorithm by using IML

Jack2012 — Sun, 18 Oct 2020 12:41:44 GMT

Thank you, Ksharp. @Rick_SAS, could you give me some help? Thank you.

Re: Optimize the algorithm by using IML

Rick_SAS — Sun, 18 Oct 2020 19:21:22 GMT

I want to be sure that I understand the situation and your request:

1. You have copied the DATA step code from the paper. The program works and gives you the correct answer.

2. Although the paper cautions not to use the program for USERN > 120, you are using it for USERN=130, so you want to find a faster way to generate the same results.

3. You think maybe converting the program to IML will make it run faster. Since you have little or no experience with IML, you are looking for someone to convert it into IML for you.

I suspect the problem is that the program is generating hundreds of millions of candidates and then writing 38.5 million records that satisfy the criterion. IMHO, converting the same algorithm into IML won't make it go substantially faster or make the final data set smaller.

Re: Optimize the algorithm by using IML

Jack2012 — Mon, 19 Oct 2020 09:00:03 GMT

Rick, thank you. Yes, your understanding is absolutely correct.

OK, then I will not try anymore.
Thank you, and Ksharp.