topic Stopping rule for chi-squared discretization algorithm in Statistical Procedures

Stopping rule for chi-squared discretization algorithm

Tenno1 — Fri, 28 Oct 2011 14:41:55 GMT

I developed an algorithm that uses the chi-squared test to perform supervised discretization of a continuous variable. I described it in the paper "ChiD-A Chi-Squared Discretization Algorithm" published in the WUSS 2011 Proceedings available at http://www.wuss.org/proceedings11/

The stopping criterion is not very intelligent, and I would like to know if there are better ways of stopping the discretization process.

Stopping rule for chi-squared discretization algorithm

art297 — Fri, 28 Oct 2011 15:04:46 GMT

I should begin by admitting that I am not a statistician and am not familiar with either the method you are using or with IML.

That said, when I have confronted situations where I needed to incorporate a somewhat intelligent stopping point, I found it useful to apply a rather brute force approach, namely to wrap the code within a macro that uses a binary decision tree to test various criteria until an acceptible limit is reached.

Of course, if you are asking what such a criterion might be, please just ignore this post.

Re: Stopping rule for chi-squared discretization algorithm

Rick_SAS — Fri, 28 Oct 2011 15:22:21 GMT

Probably a good general approach. In IML you don't even need to use a macro. In IML you can just wrap the code in a module definition and call the module at each step of an iterative method.

For example, if you're trying to find a zero, see http://blogs.sas.com/content/iml/2011/08/03/finding-the-root-of-a-univariate-function/ Or, if you're trying to optimize some criterion, see http://blogs.sas.com/content/iml/2011/10/12/maximum-likelihood-estimation-in-sasiml/