topic Re: Procs HPMixed and HPLMixed in Statistical Procedures

Procs HPMixed and HPLMixed

davidrutley — Tue, 17 May 2016 13:35:28 GMT

To Whom It May Concern;

I use SAS 9.4 and am having difficulty running a medium to large analysis. I would like to undertake a hazard analysis of some historical commercial, not experimental, animal data.

Hazard is whether a hazard was observed, this is a binary variable with a mean of 0.93 and standard dev of 0.29.

I have just under 1E6 observations on 1E5 animals.

I have

10 fixed factors

43 continuous variables

3 random factors

Having fitted a fixed effects model in Proc GLM there are many interactions and quadratic effects that are significant at P<0.0001

I have chosen to fit fixed effects and interactions, including quadratic effects only if they are significant at P<=0.0001. After correction for Nelder's marginality requirements (1994) this gives me a fixed effects model with >4,000 df.

The data is unbalanced commercial data, similar to that used for animal genetic analyses, but with a much more complicated fixed effects model.

I would like to fit animal, handler and year as random effects without any interactions.

Using HPL Mixed I receive the following error

NOTE: The HPLMIXED procedure is executing in single-machine mode.
ERROR: Insufficient resources to proceed. Terminating the mixed modeling task.

Using HP Mixed I receive the following error

     NOTE: 56969 observations are excluded because of: missing response values
    (n=377), missing fixed effects (n=56969), missing random effects
     (n=6546).
     ERROR: The MMEq matrix is not positive semidefinite.

My computer is a PC running Windows7 Professional with 64Gb RAM running at 3.6GHz.

It has 2 by 1 Tb hard disc drives that back up to one another.

I have a solid state 500Gb Hard Drive that has been included to use as virtual memory.

I am not sure what resources I need to continue with HPLMixed. It is possible that I am not directing the working directory to the solid state hard drive correctly.

I do not know what to do to ensure I get a positive semidefinite MMEq matrix.

I would be most obliged if someone could give me some direction.

See code below

TIA

Dave

proc hplmixed data=train0;
*proc hpmixed data=train0;
*P<=0.0001;
class Month doweek notwrk Site Observn L14 T6 T7 T31 T32 Handl Year Anml;
model hazd=
doweek

...

T5*T5
T9*T9
W16*W16

/ S;
random

Anml Handl Year

/ Solution;

run;

Re: Procs HPMixed and HPLMixed

Rick_SAS — Tue, 17 May 2016 14:13:29 GMT

If you haven't already read it, I recommend the 2009 paper "All the cows in Canada: Massive Mixed Modeling with the HPMIXED procedure" as a good introduction to this area. It describes a smaller problem with only 10,000 animals and many fewer effects.

Re: Procs HPMixed and HPLMixed

lvm — Tue, 17 May 2016 17:53:57 GMT

Your model is overparameterized for the fixed effects. You clearly have a lot of mixxing values (if not, then you are coding something incorrectly), but that is not an issue with mixed models. I would start with a small number of fixed effects(covariates or factors), to get things to work, and then add terms. The nonpositive definite matrix is one symptom of too many fixed effects.

HPLMIXED works best when used in multiple-machine mode. I don't use it that way, so I can't give any advice. But this requires a separate license.

You may have to increase the amount of memory that SAS can use in a job. THis is specified with the MEMSIZE option in the sas configuration file (cfg file). If this is not set at MEMSIZE=MAX, then sas will only use the memory listed in this file, often set at 2GB (it doesn't matter how much memory you actually have). I have never needed to change this option, so I hope others can tell you how to do this. You can check on this with

proc options;run;

to see what is specified.