topic Re: Change of slope in a curve in Statistical Procedures

Change of slope in a curve

BayzidurRahman — Wed, 01 Nov 2023 04:42:53 GMT

Hi,
I have a clustered count data where I have counts and denominator within each cluster for a period of 80 months. In order to examine the effect of month on the rate I fitted a random intercept Poisson model. However, the relationship is very non-linear as shown in the plot of raw rates below.

Can you please suggest how can I identify the time points at which the slope of the curve is changing (turning points)?

Re: Change of slope in a curve

Ksharp — Wed, 01 Nov 2023 12:12:55 GMT

https://blogs.sas.com/content/iml/2013/08/28/finite-diff-estimate-maxi.html

Re: Change of slope in a curve

Quentin — Wed, 01 Nov 2023 12:48:56 GMT

I don't have a suggestion, but I think your question might be slightly different.

Is your question really "how can I fit a model to this sort of data?". Meaning if you could fit a model to it that you liked, then it would be straight forward to find the points where the slope of the model changes.

As shown in the blog post linked by @Ksharp , there are model-free ways to find local maxima and minima. I think in that context for this sort of data with some noise in it, you would need to think about how to empirically define the point of a slope change aka local min/max aka vertex (?). For example, perhaps think about how many "slope change" points you see on the plot, and then think backwards to a reasonable rule that would help identify those points.

Re: Change of slope in a curve

ballardw — Wed, 01 Nov 2023 16:51:19 GMT

By "turning point" do you mean "inflection point" ? https://en.wikipedia.org/wiki/Inflection_point

Inflection point has a standard usage but I am afraid "turning point" means nothing to me.

And if "month" is the desired "effect" on rate , not "month of year", then perhaps you need to look at things differently. And if the topic is "month" why does the graph use week for tick marks???

Several of the Timeseries related procedures deal with "seasonality" such as month in different years. Perhaps that is where you need to look.

Re: Change of slope in a curve

Quentin — Wed, 01 Nov 2023 18:16:23 GMT

Perhaps I misunderstood, but I was interpreting "turning point" as being a local minimum or maximum, where the sign of the slope changes. I haven't heard this term before, but an image on the wikipedia page you linked to uses "turning point" as a label for local minima and maxima.

Edit: I found a wikipedia page with a definition for turning point, consistent with my guess: "A turning point is a point at which the derivative changes sign."

https://en.wikipedia.org/wiki/Stationary_point

Re: Change of slope in a curve

BayzidurRahman — Wed, 01 Nov 2023 19:01:47 GMT

Hi,
The term "turning point" may not be statistically appropriate. It should be "Change in trend". It is not only month but month of the year. I analysed data with both weekly and monthly resolution and by mistake I put the weekly graph. The monthly graph is given below.
During the period between 2018 and 2022 few intervention happened, whose exact timing is unclear, and they are supposed affect the trend of the rate of medication use. I need to identify those time points.

Re: Change of slope in a curve

Quentin — Wed, 01 Nov 2023 19:13:34 GMT

Can you highlight on the plot the points you would want to identify as "change in trend"? Just by eyeballing.

Re: Change of slope in a curve

BayzidurRahman — Wed, 01 Nov 2023 19:46:59 GMT

Re: Change of slope in a curve

Rick_SAS — Fri, 03 Nov 2023 13:40:00 GMT

One possible solution is to define a piecewise linear regression model.

If you know the time at which the change occurs, you can solve the problem by using a piecewise linear spline and setting the "knot locations" to be the known times. See https://blogs.sas.com/content/iml/2017/04/05/nonsmooth-models-spline-effects.html

If you do not know the change points, then you have to solve for them. These are known as segmented regression models. For an example of a single change point, see https://blogs.sas.com/content/iml/2020/12/14/segmented-regression-sas.html

Re: Change of slope in a curve

BayzidurRahman — Fri, 03 Nov 2023 13:44:56 GMT

Hi,
I do not know the time points when the changes occur and there are multiple of this. Is it possible to fit a Generalised Additive Model (GAM) to determine the smooth function and then find out the maximas and minimas by taking the derivatives?

Re: Change of slope in a curve

sbxkoenk — Fri, 03 Nov 2023 16:26:16 GMT

Paper SAS456-2017
Detecting and Adjusting Structural Breaks in Time Series and Panel Data
Using the SSM Procedure
Rajesh Selukar, SAS Institute Inc.
https://support.sas.com/resources/papers/proceedings17/SAS0456-2017.pdf
https://www.youtube.com/watch?v=7Mrmj-ERrM0
The above paper also includes a brief review of the structural break detection facilities of other SAS/ETS procedures, such as the ARIMA, AUTOREG, and UCM.
SAS/ETS Examples
Chow Test for Structural Breaks
https://support.sas.com/rnd/app/ets/examples/chow/index.htm
https://en.wikipedia.org/wiki/Chow_test
SAS/ETS® 15.1 User’s Guide
The UCM Procedure
https://support.sas.com/documentation/onlinedoc/ets/151/ucm.pdf (search term : CHECKBREAK)
Applying Data Science
by Gerhard Svolba
Chapter 6: Detecting Structural Changes in Longitudinal Data
Chapter 7 – Detecting Outliers and Level Shifts in Longitudinal Data
https://www.sas.com/storefront/aux/en/spba/63165_excerpt.pdf
I like using the ADAPTIVEREG Procedure for detecting breakpoints and structural breaks (see 6.3 !)

Br, Koen

Re: Change of slope in a curve

Rick_SAS — Fri, 03 Nov 2023 16:55:09 GMT

> Is it possible to fit a GAM to determine the smooth function and then find out the maximas and minimas by taking the derivatives?

Yes, it is possible, if that is what you want to do. If you evaluate any model (GAM, loess, spline, MARS,...) at points on a fine, evenly spaced grid for the independent variable, then you can use numerical differentiation to estimate the extrema. See Find points where a regression curve has zero slope - The DO Loop (sas.com)

Re: Change of slope in a curve

Quentin — Fri, 03 Nov 2023 16:56:46 GMT

Rajesh gave an excellent presentation to BASUG last year on PROC SSM and PRO UCM. Slides and recording are in our archives: https://www.basug.org/archives

Re: Change of slope in a curve

BayzidurRahman — Fri, 03 Nov 2023 19:08:04 GMT

Hi,
Thanks for the link. Can we find the confidence interval of the identified points?

Re: Change of slope in a curve

Rick_SAS — Fri, 03 Nov 2023 19:39:38 GMT

> Can we find a confidence interval of the identified points?

You could if you use a segmented regression model or some other rigorous statistical model. But if you fit some smoother to the data and find places where the derivative of the smoother is 0, I don't think you can get a CI automatically from the regression. For one thing, different smoothers will give you different result s. (For example, think about a loess smoother with different smoothing parameters!)

I think the fit-a-smoother-and-take-derivative method is a heuristic method that doesn't have much statistical theory behind it. If you want something like a confidence interval, you need a parameter in a statistical model.

Re: Change of slope in a curve

BayzidurRahman — Fri, 03 Nov 2023 19:59:04 GMT

What if we fit a Poisson model with fractional polynomial or spline to generate the smooth function from predicted values? There is an implementation is Stata https://minerva-access.unimelb.edu.au/items/862e275d-600b-529e-a7e8-4cf7239c1740

Re: Change of slope in a curve

Rick_SAS — Fri, 03 Nov 2023 20:12:29 GMT

Again, you could do that. It is just a different kind of smoother. But I don't think any model will enable you get CIs for the turning points.

I think you might want to discuss the problem with your colleagues or a local statistician. If this is something that you will want to publish or that has clinical implications, you would be wise to think carefully about the scientific validity of the statistical method.

If you decide on a method and program it in SAS, we can help if you run into coding difficulties.

Good luck.

Re: Change of slope in a curve

Ksharp — Mon, 13 Nov 2023 05:42:57 GMT

You could use A Time Series Decomposition to get TREND component and using Rick blog's DIFF skill to get those special kinks .

data have;
input y @@;
x+1;
cards;
5447 5412 5215 4697 4344 5426
5173 4857 4658 4470 4268 4116
4675 4845 4512 4174 3799 4847
4550 4208 4165 3763 4056 4058
5008 5140 4755 4301 4144 5380
5260 4885 5202 5044 5685 6106
8180 8309 8359 7820 7623 8569
8209 7696 7522 7244 7231 7195
8174 8033 7525 6890 6304 7655
7577 7322 7026 6833 7095 7022
7848 8109 7556 6568 6151 7453
6941 6757 6437 6221 6346 5880
;
proc iml;
use have;
read all var {y};
close ;
mdel = 0; trade = 0; year = 0;
period= 0; log = 0; maxit = 100;
update = .; /* use default update method */
line = .; /* use default line search method */
sigmax = 0; /* no upper bound for variances */
back = 100;
opt = mdel || trade || year || period || log || maxit ||
update || line || sigmax || back;
call tsdecomp(cmp,coef,aic) data=y order=2 sorder=0 nar=2
npred=5 opt=opt icmp={1 3} print=1;

trend=cmp[,1];

slope = dif(trend);
sgn = sign(slope);
difSgn = dif(sgn);

create trend var{trend difSgn};
append;
close;
quit;
data want;
merge have trend;
if difSgn in (-2 2) then kink=trend;
run;
proc sgplot data=want;
series x=x y=y/lineattrs=(color=blue);
series x=x y=trend/lineattrs=(color=red);
scatter x=x y=kink/markerattrs=(color=green size=20 symbol=starfilled);
run;

Re: Change of slope in a curve

Ksharp — Mon, 13 Nov 2023 05:57:05 GMT

Here is another function to get TREND component of time series decomposition.

Try both to see which one is better to fit your data.


data have;
 set sashelp.stocks(keep=stock date close rename=(close=y));
 if stock='IBM';
run;



proc iml;
use have;
read all var {y};
close ;
call tsbaysea(trend,season,series,adj,abic)
data=y order=2 sorder=1 npred=12 print=2;

slope = dif(trend);
sgn = sign(slope);
difSgn = dif(sgn);

create trend var{trend difSgn};
append;
close;
quit;
data want;
merge have trend;
if difSgn in (-2 2) then kink=trend;
run;
proc sgplot data=want;
series x=date y=y/lineattrs=(color=blue);
series x=date y=trend/lineattrs=(color=red);
scatter x=date y=kink/markerattrs=(color=green size=14 symbol=starfilled);
run;

Re: Change of slope in a curve

Rick_SAS — Mon, 13 Nov 2023 11:34:42 GMT

KSharp: Your two examples show the point I was trying to make: the problem is not well-defined if you define the turning points by using a smoother. As you showed, choosing different smoothers will usually result in a different number and location of the turning points.