Evaluating the Hessian

[MikeHunter]

I have been working with Proc Nlin to derive a Hessian matrix of 2nd order derivatives.  I use the OUTEST= statement to generate the statistical output and analyze it when the _TYPE_='DETERMIN' and keep the automatic variable labelled _RHS_ which contains the value for the determinant.  I don't evaluate the matrix that this determinant is derived from.

  My questions concerns the meaning of the Hessian:  its magnitude, sign, statistical significance, etc.  I can't find any literature on evaluating it...is it a standardized statistic like a t-statistic or a Z-score?  Any suggestions for tests to analyze whether or not it's significantly different from zero?

Thank you in advance.

[SteveDenham]

Hi Mike,

I think I'm missing something here. Are you asking about the determinant of the Hessian matrix?  That tells you whether the Hessian matrix is singular (=bad) or nonsingular (=good).  For most of the procs that use Hessians, you can tune this value.  The closer it is to zero, the more likely it is to be singular.

Steve Denham

[MikeHunter]

Steve-

     Thanks for contributing your point of view.  You are correct although the literature I'm familiar with also identifies the determinant as the Hessian.  And I have an extended use or interpretation for the Hessian:  if the slope is the rate of change in Y given a 1 unit change in X, the Hessian is the acceleration (deceleration) in that rate of change.  Gallant spoke of it as an index for nonlinearity.

     I understand that the smaller the value is, the more likely it is to be singular or insignificant.  I'm looking for a test that allows me to go beyond eyeballing to make a more rigorous, statistical decision.

Mike

[SteveDenham]

Mike--

There doesn't seem to be a lot of info on my bookshelf on the distributional assumptions associated with the Hessian.  I would think that to get a test you would need some sort of estimate of the error associated with the determinant, and then you could apply something like Tschebycheff's theorem.  The hard part is that there doesn't seem to even be an empirical measure of the variability, since it is the root of a polynomial.  Maybe bootstrapping could give you something.

Steve Denham

[Rick_SAS]

The Hessian is the matrix, not the determinant.

I agree with Steve that a nonparametric bootstrap is probably the best that you can do: resample many times from your data, solve the NLMIX problem that gives the determinant at the optimal value, and look at the distribution of the determinant.

However, we might be able to say more if we have more context. Often the Hessian appears when maximizing the negative log-likelihood function. At a nondegenerate optimal solution, the Hessian will be positive definite, and thus the determinant is positive 100% of the time.  At a degenerate solution, the determinant is zero.

[MoshiurRahman]

Dr. Wicklin:

I wanted to know, if please, more elaborate on your last sentence regarding the maximization and estimation. As you know inverse of hessian in the Newton iteration step does not exist in the case of degenerate solution. How can we then direct the optimization routine to proceed for a solution?

Thank you in advance.

Moshiur Rahman

[Rick_SAS]

You've asked the million-dollar question.  There are various heuristic methods that attempt to restart the optimization, but many times the real problem is that your data do not fit the model.   The textbook response is that you should choose a different model, and that would be my recommendation for you.

If you are interested in the theory, there have been many research papers written about what to do when the Hessian is singular. I recommend the 2004 paper by Jeff Gill and Gary King, "What to do when your Hessian is not invertible."  Gary King has done some excellent work in several areas of statistics, so I respect his opinions on this topic.