Finding the inflection point for best threshold

[harmonic]

Hello community,

I would like to find the best threshold with a function like that:

For now I used the slop function to find the sign differences:

data sorted_QUANTILI_DS;
set sorted_QUANTILI_DS;
slope = dif(prob_nozero_cum) / dif(index); /* estimate the slope by using backward difference */
SlopeChanged = dif(slope); /* indicate when the slope changes */
run;

Is there a way to use the call nlpfdd based on this article and use the finite-differnece derivatives:
Finite-difference derivatives in SAS - The DO Loop


Thank you in advance

[Rick_SAS]

Yes, assuming that you have a formula for the function. If so, the inflection points are the values of x where the graph changes from concave up to concave down or vice versa. This requires finding a point where the second derivative changes sign, which means you can use the NLPFDD function to obtain numerical second derivatives. You can use the FROOT function in PROC IML to 

find the numerical roots of any 1-D function on a specified interval. In the following, I find the inflection point for the function sin(x) when x is in the interval [pi,2, 3*pi/2]:

proc iml;
/* define the function */
start Func(x);
return( sin(x) );
finish;

/* define function that finds the numerical second derivative of 'Func' */
start SecondDeriv(x);
call nlpfdd(fun, deriv, second_deriv, 'Func', x);
return( second_deriv );
finish;

/* find the inflection point for Func on the interval [a,b] */
pi = constant('pi');
a = pi/2;
b = 3*pi/2;
infl_pt = froot("SecondDeriv", a||b );
print infl_pt;

For your example, the interval would be {0, 50}. You would replace the definition of the FUNC function with your function. 

If you don't have a formula for the function, the problem gets much harder. You have to use some sort of interpolation scheme to approximate the function, then find the inflection point of the interpolant.

[harmonic]

Yes I do not have the form of the function, just the interval and the y. Do you have any advice for interpolation?

[Rick_SAS]

Sure. You say that you want a numerical estimate of the inflection point based only on knowing the function at a finite set of points (x[i], f(x[i])).

Here's what I suggest.

1. Use a finite-difference formula to approximate the second derivative of the function. I'd suggest using central differences for 2nd derivatives. Formulas are at
https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/imlug/imlug_nonlinearoptexpls_sect012.htm

You will get a vector (call if D2F) that for each value of x estimates the second derivative at x.

2. Find the location(s) where the second derivative changes sign.

3. Let i-1 and i be the indices for which D2F have different sign. Let xL = x[i-1] and xR = x[i], and let yL = D2F[i-1] and yR = D2f[i]. Then you can define 
m = (yR - yL) / (xR - xL);

inflectionPt = xL - yL / m;

[Rick_SAS · Accepted solution]

In summary, here are two references for how to numerically estimate inflection points:

1. Find an inflection point for a function when you know a formula (al algorithm) for the function

2. Find an inflection point when you know the function only at a finite set of points.

From what you've said, the second item is relevant to your problem.