topic Re: How to fit data combing a truncated normal distribution with a weibull distribution in Statistical Procedures

How to fit data combing a truncated normal distribution with a weibull distribution

JamesXu — Fri, 14 Jun 2013 07:34:21 GMT

Hello everyone:

As I know, there are some functions fitting normal distribution, Weibull distribution, or their combination in the FMM procedure.

However, if I want to fit data which obey a truncated normal distribution combining with a Weibull distribution, such as log-transformed time intervals during feeding in chickens (the simulative image is attached), how to do it?

It will be highly appreciated if the SAS code is given.

Thanks for your kind help and looking forward your quick reply!

Best regards!

Re: How to fit data combing a truncated normal distribution with a weibull distribution

SteveDenham — Fri, 14 Jun 2013 12:31:12 GMT

Not sure but, Example.37.2 The Usefulness of Custom Starting Values: When Do Cows Eat? really looks like a good starting place.

I think you will need the counts rather than the proportions shown on the jpeg, but that should not be a problem.

proc fmm data=yourdata;

model logint = / dist=truncnormal;

model + / dist=weibull;

freq count;

run;

I hope this can get you started.

Steve Denham

Message was edited by: Steve Denham

Re: How to fit data combing a truncated normal distribution with a weibull distribution

JamesXu — Sun, 16 Jun 2013 04:25:51 GMT

Dear Steve:

Thank you for your kind and professional reply！

Example.37.2 “The Usefulness of Custom Starting Values: When Do Cows Eat?” is indeed a good starting place. Taking it as an example, supposing the entered original data are unchanged (including logInt and relevant counts), if I only want to fit the posterior part of the data after the x-axis point 7.9 (clear part of simulative image 2), using only a truncated (truncation=7.9) normal distribution regardless of the distribution pattern for the anterior part (obscure part of simulative image2), what can I do?

I use the below SAS code to obtain relative info:

proc fmm data=cattle;

model logint = / dist=truncnormal(7.9,.);

freq count;

run;

But it doesn't work, following by the warning info :"Adequate starting values could not be determined for this model. The model is probably over-parameterized and contains too many components."

Does my code wrong?

Re: How to fit data combing a truncated normal distribution with a weibull distribution

JamesXu — Sun, 16 Jun 2013 06:08:22 GMT

Re: How to fit data combing a truncated normal distribution with a weibull distribution

SteveDenham — Mon, 17 Jun 2013 13:24:40 GMT

I don't think your code is incorrect. When I submitted it (SAS 9.3 (TS1M2) on a 64 bit Windows 7 machine), it worked fine, converging in six iterations, to an intercept estimate = 9.2544 and scale estimate = 0.4453.

I would check the version of SAS you are using, and if it is the same as mine, I think you need to contact Technical Support.

Steve Denham

Re: How to fit data combing a truncated normal distribution with a weibull distribution

JamesXu — Sat, 22 Jun 2013 01:14:41 GMT

Dear steve:

I always run SAS 9.3 on a 64 bit Windows XP machine, which is the same as yours! When I installed this 9.3 version, some lauguage module was ignored, but I don't think this impair the operation process! The code that mentioned previously does't work despite I have submitted it many times. Could you pls copy the right code to this forum or my email ( zhenqiangxu@163.com)?

Take example 37.2 as an example again, supposed I can obtain a truncated normal model, the PDF of this truncated normal :

where: cunormal = cumulative normal correction factor to allow for truncation, m and s = mean and standard deviation of the interval lengths distribution and x = logint, 7.5 is the point for truncation.

What I really concern is that how can I get the point at which the PDF accounted for exactly half of the observed frequency?

Maybe the FMM procedure is not the unique and best way to obtain what I need, so Mr. Steve, could you help me to solve this crucial problem using either other basic codes or the FMM code?

Thanks a lot and best regards!

Re: How to fit data combing a truncated normal distribution with a weibull distribution

SteveDenham — Mon, 24 Jun 2013 13:15:32 GMT

James,

I submitted the following:

data cattle;

input LogInt Count @@;

datalines;

0.70 195 1.10 233 1.40 355 1.60 563

1.80 822 1.95 926 2.10 1018 2.20 1712

2.30 3190 2.40 2212 2.50 1692 2.55 1558

2.65 1622 2.70 1637 2.75 1568 2.85 1599

2.90 1575 2.95 1526 3.00 1537 3.05 1561

3.10 1555 3.15 1427 3.20 2852 3.25 1396

3.30 1343 3.35 2473 3.40 1310 3.45 2453

3.50 1168 3.55 2300 3.60 2174 3.65 2050

3.70 1926 3.75 1849 3.80 1687 3.85 2416

3.90 1449 3.95 2095 4.00 1278 4.05 1864

4.10 1672 4.15 2104 4.20 1443 4.25 1341

4.30 1685 4.35 1445 4.40 1369 4.45 1284

4.50 1523 4.55 1367 4.60 1027 4.65 1491

4.70 1057 4.75 1155 4.80 1095 4.85 1019

4.90 1158 4.95 1088 5.00 1075 5.05 912

5.10 1073 5.15 803 5.20 924 5.25 916

5.30 784 5.35 751 5.40 766 5.45 833

5.50 748 5.55 725 5.60 674 5.65 690

5.70 659 5.75 695 5.80 529 5.85 639

5.90 580 5.95 557 6.00 524 6.05 473

6.10 538 6.15 444 6.20 456 6.25 453

6.30 374 6.35 406 6.40 409 6.45 371

6.50 320 6.55 334 6.60 353 6.65 305

6.70 302 6.75 301 6.80 263 6.85 218

6.90 255 6.95 240 7.00 219 7.05 202

7.10 192 7.15 180 7.20 162 7.25 126

7.30 148 7.35 173 7.40 142 7.45 163

7.50 152 7.55 149 7.60 139 7.65 161

7.70 174 7.75 179 7.80 188 7.85 239

7.90 225 7.95 213 8.00 235 8.05 256

8.10 272 8.15 290 8.20 320 8.25 355

8.30 307 8.35 311 8.40 317 8.45 335

8.50 369 8.55 365 8.60 365 8.65 396

8.70 419 8.75 467 8.80 468 8.85 515

8.90 558 8.95 623 9.00 712 9.05 716

9.10 829 9.15 803 9.20 834 9.25 856

9.30 838 9.35 842 9.40 826 9.45 834

9.50 798 9.55 801 9.60 780 9.65 849

9.70 779 9.75 737 9.80 683 9.85 686

9.90 626 9.95 582 10.00 522 10.05 450

10.10 443 10.15 375 10.20 342 10.25 285

10.30 254 10.35 231 10.40 195 10.45 186

10.50 143 10.55 100 10.60 73 10.65 49

10.70 28 10.75 36 10.80 16 10.85 9

10.90 5 10.95 6 11.00 4 11.05 1

11.15 1 11.25 4 11.30 2 11.35 5

11.40 4 11.45 3 11.50 1

;

proc fmm data=cattle gconv=1e-12;

model logint = / dist=truncnormal(7.9,.);

freq count;

run;

Convergence was obtained in six iterations with the following output

Parameter Estimates for 'Truncated Normal' Model

Effect Estimate Standard Error z Value Pr > |z|

Intercept 9.2544 0.004415 2096.11 <.0001

Scale Parameter 0.4453 0.004623

If your installation is not giving you this, contact Technical Support, as there is likely something wrong in your installation.

Steve Denham

Re: How to fit data combing a truncated normal distribution with a weibull distribution

lvm — Mon, 24 Jun 2013 14:25:38 GMT

I think there are several bugs in the early versions of FMM. Steve, I think you have STAT 12.1 with your 9.3? I still don't have 12.1, and the example program also bombs on my machine. I have found other problems with the experimental FMM. I bet these are all fixed in 12.1.

Re: How to fit data combing a truncated normal distribution with a weibull distribution

SteveDenham — Mon, 24 Jun 2013 14:37:36 GMT

Yep. I've got 12.1 on my PC (and am looking forward to STAT12.3 and SAS9.4). And not looking forward to two stage upgrades on servers still on 9.1.3...

Steve Denham

Re: How to fit data combing a truncated normal distribution with a weibull distribution

Rick_SAS — Tue, 25 Jun 2013 14:50:51 GMT

You can estiamte the 0.5 quantile ("the point at which the PDF accounted for exactly half of the observed frequency") in several ways, not all of which involve PROC FMM:

1) Find the sample median by using PROC MEANS of PROC UNIVARIATE

2) If you really want to use a density estimate, use the HISTOGRAM stmt of PROC UNIVARIATE and the OUTKERNEL= option to get the nonparametric KDE in a data set. Then numerically integrate it, as shoiwn in this article: The area under a density estimate curve: Nonparametric estimates - The DO Loop

3) For a parametric approach, fit the parameters of your truncnormal+weibull model and then use the CDF function. If f = a*f1 + (1-a)*f2, then cdf(f) = a*cdf(f1) + (1-a)*cdf(f2). The CDF of the truncated normal can be computed from the CDF of the normal (see Wikipedia) and the Weibull distribution is supported by the SAS CDF function directly.