turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

10-20-2017 02:33 PM

Hi all.

I'm using Warren Kuhfeld's marvellous experimental design macro, %choiceff() to find large choice designs from even larger candidate data sets, and when I ramp up to 16 alternatives and their 2-way cross availability effects + generic price I have 251 parameters in the design.

```
%let nupc=16;
%macro mk2clst(mynalts=);
%do i = 1 %to &mynalts.;
%do j = 1 %to &mynalts.;
%if
%eval(&i. NE &j.)
%then
%do;
c%trim(%eval(%eval(%eval(&i.-1)*&mynalts.)+&j.))
%end;
%end;
%end;
%mend;
%choiceff
(
data= perm.yogurt,
model= class(a1 - a%trim(&nupc.) %mk2clst(mynalts=&nupc.) / zero=first sta) identity(price),
nsets= %eval(144*32),
nalts= %trim(&nupc.),
seed= 09051958,
options= nodups,
rscale= alt,
maxiter= 1,
beta= zero
)
;
```

The macro has to evaluate the determinant as that is part of the all-important design D-efficiency statistic. This interesting thing about my error is that the problem size of 251 parameters completes normally when I run it inside the latest SAS University Edition VM (SAS 9.4 m4 64 bit Centos) , but it terminates with the error in the title when I run exactly the same analysis on my win x64 desktop SAS 9.4 M1. Is this just another 'linux is better than windows' example? I note that in other community posts Rick_SAS has explained that this error arises when the determinant of a large matrix exceeds the double precision floating point representation limit. I'm planning to scale up my analysis to 175 alternatives (!) so I guess I'll have to re-write the determinant calculation using Rick_SAS's suggestion of calculating log(det), then scale the log det term before exponentiating. Attempts to find designs with 17 alternatives and their 2-way cross availability effects fail in both environments with less informative errors but I assume the underlying problem is the same - matrix determinant too big for double floating point representation. Has anyone else struck this limit with Warren's %choiceff() macro? Any tips before I dive down that rabbit hole?

Accepted Solutions

Solution

10-20-2017
10:59 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Damien_Mather

10-20-2017 03:01 PM

Hi Damian,

Thanks for the kind words! I requested a log determinant function in IML quite a few years back. It was delivered in 9.4M1. I have been using it ever since in several of my macros including ChoicEff. Are you sure that you are using a 9.4M1 version of the macro?

In case you are interested, here is how I get efficiency.

```
%macro _logdet(logdet, eff, xpx, npi);
holdlogdet = logabsdet(&xpx);
if holdlogdet[2] <= 0 then holdlogdet[1] = -1e50;
%if &logdet ne %then %do; &logdet = holdlogdet[1]; %end;
%if &eff ne %then %do; &eff = exp(holdlogdet[1] # &npi); %end;
%mend;
```

Best,

Warren

All Replies

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Damien_Mather

10-20-2017 03:00 PM

The LOGABSDET function was introduced in SAS 9.4M1. That was a long time ago, but if I recall correctly, I mentioned the LOGABSDET function to @WarrenKuhfeld sometime later and he updated his macros to use it. If so, it might not be a Window/Linux issue, but that the 9.4M1 macro uses different computations than the 9.4M4 macro.

Warren might chime in if he has something to say.

Solution

10-20-2017
10:59 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Damien_Mather

10-20-2017 03:01 PM

Hi Damian,

Thanks for the kind words! I requested a log determinant function in IML quite a few years back. It was delivered in 9.4M1. I have been using it ever since in several of my macros including ChoicEff. Are you sure that you are using a 9.4M1 version of the macro?

In case you are interested, here is how I get efficiency.

```
%macro _logdet(logdet, eff, xpx, npi);
holdlogdet = logabsdet(&xpx);
if holdlogdet[2] <= 0 then holdlogdet[1] = -1e50;
%if &logdet ne %then %do; &logdet = holdlogdet[1]; %end;
%if &eff ne %then %do; &eff = exp(holdlogdet[1] # &npi); %end;
%mend;
```

Best,

Warren

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to WarrenKuhfeld

10-20-2017 11:11 PM

Thanks for the explanations, Warren and Rick. You guys are the best. I checked and indeed, an older version had been re-installed on my desktop in error during an automated group software update programmed by my IT support staff. My bad for not checking the startup log! Back to the future (tm)? Forward to the past? Anyway,, I'm sure when they eventually install the latest version, you fantastic macros will all run just perfectly.

Warren, with the log det improvements since 9.4 M1, have you or other %choiceff() users encountered an upper parameter limit for designs? Can we now all reach out into the (big data) universe for our design-balanced choice model data?

Warren, with the log det improvements since 9.4 M1, have you or other %choiceff() users encountered an upper parameter limit for designs? Can we now all reach out into the (big data) universe for our design-balanced choice model data?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Damien_Mather

10-21-2017 07:10 AM

Damien,

I emailed you the 9.4M1 version. I quit putting new versions on the web since we update SAS so often now days. Email me if you need a different version. There are no built in limitations on the size of the problem, but this is a highly iterative process, and run time does not scale linearly with problem size. I have found that usually you get most of the efficiency gain in the first iteration. Don't ask for a lot of iterations with a big problem.

Best,

Wrren

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to WarrenKuhfeld

10-20-2017 11:14 PM - edited 10-21-2017 12:04 AM

I guess in the meantime, while I wait for my IT support staff, I could just download the 9.4 M4 mr macros to overwrite the older versions, right?

Here is the code I use to copy the latest version of choiceff.sas from inside my current SAS University Edition:

```
filename macrout '/folders/myfolders/mymacros/';
data _null_;
infile sasautos(choiceff.sas);
file macrout(choiceff.sas);
input ;
put _infile_;
run;
```

and from there, a shared folder with my windows host operating system, I can copy it to overwrite my existing SAS 9.4 M0 for win for x64 supplied macro.