BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
Damien_Mather
Lapis Lazuli | Level 10

Hi all.

 

I'm using Warren Kuhfeld's marvellous experimental design macro, %choiceff() to find large choice designs from even larger candidate data sets, and when I ramp up to 16 alternatives and their 2-way cross availability effects + generic price I have 251 parameters in the design.

%let nupc=16;

%macro mk2clst(mynalts=);
 %do i = 1 %to &mynalts.;
  %do j = 1 %to &mynalts.;
   %if
    %eval(&i. NE &j.)
   %then
    %do;
     c%trim(%eval(%eval(%eval(&i.-1)*&mynalts.)+&j.)) 
    %end; 
  %end;
 %end;
%mend;

%choiceff
   (
    data= perm.yogurt,
    model= class(a1 - a%trim(&nupc.) %mk2clst(mynalts=&nupc.) / zero=first sta) identity(price), 
    nsets= %eval(144*32),
    nalts= %trim(&nupc.),
    seed= 09051958,
    options= nodups,
	rscale= alt,
    maxiter= 1,
    beta= zero
   )
  ; 

 

The macro has to evaluate the determinant as that is part of the all-important design D-efficiency statistic. This interesting thing about my error is that the problem size of 251 parameters completes normally when I run it inside the latest SAS University Edition VM (SAS 9.4 m4 64 bit Centos) , but it terminates with the error in the title when I run exactly the same analysis on my win x64 desktop SAS 9.4 M1. Is this just another 'linux is better than windows' example? I note that in other community posts Rick_SAS has explained that this error arises when the determinant of a large matrix exceeds the double precision floating point representation limit. I'm planning to scale up my analysis to 175 alternatives (!) so I guess I'll have to re-write the determinant calculation using Rick_SAS's suggestion of calculating log(det), then scale the log det term before exponentiating. Attempts to find designs with 17 alternatives and their 2-way cross availability effects fail in both environments with less informative errors but I assume the underlying problem is the same - matrix determinant too big for double floating point representation. Has anyone else struck this limit with Warren's %choiceff() macro? Any tips before I dive down that rabbit hole?

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
WarrenKuhfeld
Rhodochrosite | Level 12

Hi Damian,
Thanks for the kind words!  I requested a log determinant function in IML quite a few years back.  It was delivered in 9.4M1.  I have been using it ever since in several of my macros including ChoicEff.  Are you sure that you are using a 9.4M1 version of the macro?

In case you are interested, here is how I get efficiency.

 

%macro _logdet(logdet, eff, xpx, npi);
   holdlogdet = logabsdet(&xpx);
   if holdlogdet[2] <= 0 then holdlogdet[1] = -1e50;
   %if &logdet ne %then %do; &logdet = holdlogdet[1];             %end;
   %if &eff    ne %then %do; &eff    = exp(holdlogdet[1] # &npi); %end;
   %mend;

Best,

Warren

View solution in original post

5 REPLIES 5
Rick_SAS
SAS Super FREQ

The LOGABSDET function was introduced in SAS 9.4M1. That was a long time ago, but if I recall correctly, I mentioned the LOGABSDET function to @WarrenKuhfeld sometime later and he updated his macros to use it. If so, it might not be a Window/Linux issue, but that the 9.4M1 macro uses different computations than the 9.4M4 macro.

 

Warren might chime in if he has something to say.

WarrenKuhfeld
Rhodochrosite | Level 12

Hi Damian,
Thanks for the kind words!  I requested a log determinant function in IML quite a few years back.  It was delivered in 9.4M1.  I have been using it ever since in several of my macros including ChoicEff.  Are you sure that you are using a 9.4M1 version of the macro?

In case you are interested, here is how I get efficiency.

 

%macro _logdet(logdet, eff, xpx, npi);
   holdlogdet = logabsdet(&xpx);
   if holdlogdet[2] <= 0 then holdlogdet[1] = -1e50;
   %if &logdet ne %then %do; &logdet = holdlogdet[1];             %end;
   %if &eff    ne %then %do; &eff    = exp(holdlogdet[1] # &npi); %end;
   %mend;

Best,

Warren

Damien_Mather
Lapis Lazuli | Level 10
Thanks for the explanations, Warren and Rick. You guys are the best. I checked and indeed, an older version had been re-installed on my desktop in error during an automated group software update programmed by my IT support staff. My bad for not checking the startup log! Back to the future (tm)? Forward to the past? Anyway,, I'm sure when they eventually install the latest version, you fantastic macros will all run just perfectly.

Warren, with the log det improvements since 9.4 M1, have you or other %choiceff() users encountered an upper parameter limit for designs? Can we now all reach out into the (big data) universe for our design-balanced choice model data?
WarrenKuhfeld
Rhodochrosite | Level 12

Damien,

I emailed you the 9.4M1 version.  I quit putting new versions on the web since we update SAS so often now days. Email me if you need a different version.  There are no built in limitations on the size of the problem, but this is a highly iterative process, and run time does not scale linearly with problem size.  I have found that usually you get most of the efficiency gain in the first iteration.  Don't ask for a lot of iterations with a big problem.

Best,

Wrren

Damien_Mather
Lapis Lazuli | Level 10

I guess in the meantime, while I wait for my IT support staff, I could just download the 9.4 M4 mr macros to overwrite the older versions, right?

 

Here is the code I use to copy the latest version of choiceff.sas from inside my current SAS University Edition:

 

filename macrout '/folders/myfolders/mymacros/';
data _null_;
 infile sasautos(choiceff.sas);
 file macrout(choiceff.sas);
input ;
put _infile_; 
run;

and from there, a shared folder with my windows host operating system, I can copy it to overwrite my existing SAS 9.4 M0 for win for x64 supplied macro.

 

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

Multiple Linear Regression in SAS

Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.

Find more tutorials on the SAS Users YouTube channel.

From The DO Loop
Want more? Visit our blog for more articles like these.
Discussion stats
  • 5 replies
  • 1397 views
  • 5 likes
  • 3 in conversation