Your model is overparameterized for the fixed effects. You clearly have a lot of mixxing values (if not, then you are coding something incorrectly), but that is not an issue with mixed models. I would start with a small number of fixed effects(covariates or factors), to get things to work, and then add terms. The nonpositive definite matrix is one symptom of too many fixed effects.
HPLMIXED works best when used in multiple-machine mode. I don't use it that way, so I can't give any advice. But this requires a separate license.
You may have to increase the amount of memory that SAS can use in a job. THis is specified with the MEMSIZE option in the sas configuration file (cfg file). If this is not set at MEMSIZE=MAX, then sas will only use the memory listed in this file, often set at 2GB (it doesn't matter how much memory you actually have). I have never needed to change this option, so I hope others can tell you how to do this. You can check on this with
proc options;run;
to see what is specified.
... View more