Statistical programming, matrix languages, and more

%choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win x64

Accepted Solution Solved
Reply
Frequent Contributor
Posts: 136
Accepted Solution

%choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win x64

Hi all.

 

I'm using Warren Kuhfeld's marvellous experimental design macro, %choiceff() to find large choice designs from even larger candidate data sets, and when I ramp up to 16 alternatives and their 2-way cross availability effects + generic price I have 251 parameters in the design.

%let nupc=16;

%macro mk2clst(mynalts=);
 %do i = 1 %to &mynalts.;
  %do j = 1 %to &mynalts.;
   %if
    %eval(&i. NE &j.)
   %then
    %do;
     c%trim(%eval(%eval(%eval(&i.-1)*&mynalts.)+&j.)) 
    %end; 
  %end;
 %end;
%mend;

%choiceff
   (
    data= perm.yogurt,
    model= class(a1 - a%trim(&nupc.) %mk2clst(mynalts=&nupc.) / zero=first sta) identity(price), 
    nsets= %eval(144*32),
    nalts= %trim(&nupc.),
    seed= 09051958,
    options= nodups,
	rscale= alt,
    maxiter= 1,
    beta= zero
   )
  ; 

 

The macro has to evaluate the determinant as that is part of the all-important design D-efficiency statistic. This interesting thing about my error is that the problem size of 251 parameters completes normally when I run it inside the latest SAS University Edition VM (SAS 9.4 m4 64 bit Centos) , but it terminates with the error in the title when I run exactly the same analysis on my win x64 desktop SAS 9.4 M1. Is this just another 'linux is better than windows' example? I note that in other community posts Rick_SAS has explained that this error arises when the determinant of a large matrix exceeds the double precision floating point representation limit. I'm planning to scale up my analysis to 175 alternatives (!) so I guess I'll have to re-write the determinant calculation using Rick_SAS's suggestion of calculating log(det), then scale the log det term before exponentiating. Attempts to find designs with 17 alternatives and their 2-way cross availability effects fail in both environments with less informative errors but I assume the underlying problem is the same - matrix determinant too big for double floating point representation. Has anyone else struck this limit with Warren's %choiceff() macro? Any tips before I dive down that rabbit hole?

 

 


Accepted Solutions
Solution
‎10-20-2017 10:59 PM
SAS Super FREQ
Posts: 496

Re: %choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win

Posted in reply to Damien_Mather

Hi Damian,
Thanks for the kind words!  I requested a log determinant function in IML quite a few years back.  It was delivered in 9.4M1.  I have been using it ever since in several of my macros including ChoicEff.  Are you sure that you are using a 9.4M1 version of the macro?

In case you are interested, here is how I get efficiency.

 

%macro _logdet(logdet, eff, xpx, npi);
   holdlogdet = logabsdet(&xpx);
   if holdlogdet[2] <= 0 then holdlogdet[1] = -1e50;
   %if &logdet ne %then %do; &logdet = holdlogdet[1];             %end;
   %if &eff    ne %then %do; &eff    = exp(holdlogdet[1] # &npi); %end;
   %mend;

Best,

Warren

View solution in original post


All Replies
SAS Super FREQ
Posts: 4,019

Re: %choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win

Posted in reply to Damien_Mather

The LOGABSDET function was introduced in SAS 9.4M1. That was a long time ago, but if I recall correctly, I mentioned the LOGABSDET function to @WarrenKuhfeld sometime later and he updated his macros to use it. If so, it might not be a Window/Linux issue, but that the 9.4M1 macro uses different computations than the 9.4M4 macro.

 

Warren might chime in if he has something to say.

Solution
‎10-20-2017 10:59 PM
SAS Super FREQ
Posts: 496

Re: %choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win

Posted in reply to Damien_Mather

Hi Damian,
Thanks for the kind words!  I requested a log determinant function in IML quite a few years back.  It was delivered in 9.4M1.  I have been using it ever since in several of my macros including ChoicEff.  Are you sure that you are using a 9.4M1 version of the macro?

In case you are interested, here is how I get efficiency.

 

%macro _logdet(logdet, eff, xpx, npi);
   holdlogdet = logabsdet(&xpx);
   if holdlogdet[2] <= 0 then holdlogdet[1] = -1e50;
   %if &logdet ne %then %do; &logdet = holdlogdet[1];             %end;
   %if &eff    ne %then %do; &eff    = exp(holdlogdet[1] # &npi); %end;
   %mend;

Best,

Warren

Frequent Contributor
Posts: 136

Re: %choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win

Posted in reply to WarrenKuhfeld
Thanks for the explanations, Warren and Rick. You guys are the best. I checked and indeed, an older version had been re-installed on my desktop in error during an automated group software update programmed by my IT support staff. My bad for not checking the startup log! Back to the future (tm)? Forward to the past? Anyway,, I'm sure when they eventually install the latest version, you fantastic macros will all run just perfectly.

Warren, with the log det improvements since 9.4 M1, have you or other %choiceff() users encountered an upper parameter limit for designs? Can we now all reach out into the (big data) universe for our design-balanced choice model data?
SAS Super FREQ
Posts: 496

Re: %choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win

Posted in reply to Damien_Mather

Damien,

I emailed you the 9.4M1 version.  I quit putting new versions on the web since we update SAS so often now days. Email me if you need a different version.  There are no built in limitations on the size of the problem, but this is a highly iterative process, and run time does not scale linearly with problem size.  I have found that usually you get most of the efficiency gain in the first iteration.  Don't ask for a lot of iterations with a big problem.

Best,

Wrren

Frequent Contributor
Posts: 136

Re: %choiceff() macro searching for 251 parm designs halts with error: overflow in DET error on win

[ Edited ]
Posted in reply to WarrenKuhfeld

I guess in the meantime, while I wait for my IT support staff, I could just download the 9.4 M4 mr macros to overwrite the older versions, right?

 

Here is the code I use to copy the latest version of choiceff.sas from inside my current SAS University Edition:

 

filename macrout '/folders/myfolders/mymacros/';
data _null_;
 infile sasautos(choiceff.sas);
 file macrout(choiceff.sas);
input ;
put _infile_; 
run;

and from there, a shared folder with my windows host operating system, I can copy it to overwrite my existing SAS 9.4 M0 for win for x64 supplied macro.

 

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 337 views
  • 5 likes
  • 3 in conversation