topic Re: Normality Test in Statistical Procedures

Normality Test

Babloo — Mon, 02 Feb 2015 13:49:55 GMT

I've used the univariate procedure to determine the normality for the continuous varaible 'amount'. With the actual data mean is 5055 and the median is 68. Similarly skewness and kutosis is 8.5 and 166 respectively.

proc univariate data=want;

var amount

histogram /normal (color=red);

run;

As per the documentation, I understood that mean and median should almost remains the same (both values should be close to each other) and skewness and kurtosis should be close to '0' for the normal curve.So do I need to remove the outliers to make my data normal? Or we've any other better solution to create a normal data?

What other procedures\techniques can be used in SAS to conduct a normality test?

Thanks for any help you offer.

Re: Normality Test

ballardw — Mon, 02 Feb 2015 15:27:36 GMT

On the proc univariate statement add the option NORMAL. This will add some output that is the result of tests for normality. Removing outliers, if any, is a BIG topic in analysis.

Re: Normality Test

SteveDenham — Mon, 02 Feb 2015 17:51:09 GMT

I would expect that the variable 'amount' has a lognormal distribution.

Try doing a log transformation of your data, and then look at the various moments.

Steve Denham

Re: Normality Test

Babloo — Tue, 03 Feb 2015 08:05:22 GMT

I don't see much difference in my output with the options 'Normaltest'. My code is below.

proc univariate data=normality normaltest ;

var balance;

histogram /normal (color=red);

run;

Any other suggestions to produce a normal data?

Thanks!

Re: Normality Test

Babloo — Tue, 03 Feb 2015 12:25:52 GMT

Minimum account in my dataset is 0 and the maximum is 124346. When I tried the code below.

ods graphics on;

options maxmemquery =6M;

proc univariate data=Anova_data_new normaltest plots;

var tot_balanc;

histogram /midpoints=1 to 124346 by 1000 /*How to find the divisor value*/

lognormal;

run;

ods graphics off;

Encounted by error as ERROR: The smallest value of amount is less than or equal to the threshold parameter (THETA) for the lognormal fit. According to documenation,

The threshold parameter must be less than the minimum data value. You can specify with the THRESHOLD= lognormal-option. By default, . If you specify THETA=EST, a maximum likelihood estimate is computed for . You can specify and with the SCALE= and SHAPE= lognormal-options, respectively. By default, the procedure calculates maximum likelihood estimates for these parameters.

But I'm not sure how to compute theta value for my data. May I request you to extend your help on the same? Also let me know why we going for log normal distribution instead of normal distribution?

Re: Normality Test

Babloo — Wed, 04 Feb 2015 07:33:20 GMT

Any suggestions?

Re: Normality Test

gergely_batho — Wed, 04 Feb 2015 11:04:29 GMT

Hi,

It is true, that for a normal distribution "mean and median should almost remains the same (both values should be close to each other) and skewness and kurtosis should be close to 0". But there are formal statistical tests of normality, which are available in proc univariate.

Obviously your data is not normally distributed, this is why Steve suggested to test lognormality of the data.

A log normally distributed data does not contain 0-s. But maybe it doesn’t hurt to add 1 to those 0 values. Alternatively you can apply threshold=-1 (shifting the fitted distribution).

Or you could try other distributions as well.

Why is lognormal better than normal? Because (probably) it fits your data better.

But we don’t know the purpose of your study.

Why do you want to fit a distribution to your data? Why do you want to test normality (or lognormality or something else) of your data? Why do you want to make it normal? (By removing outliers maybe.)

Gergely

Re: Normality Test

Babloo — Wed, 04 Feb 2015 11:19:31 GMT

Thanks for your valuable comments.

I need to conduct a ANOVA test with a samples. Hence I would except my samples to be normal.

Will it sounds good if I remove the values which are less than or equal 0 before applying log-normal distribution ? I wonder when I see the various moments (non-normal to normal) for my variable after I switch over from normal to log-normal? What is the significance behind log-normal distribution? Can it be applied to the variable 'balance/amount'?

Re: Normality Test

gergely_batho — Wed, 04 Feb 2015 11:40:15 GMT

You have different groups, and you want to determine whether mean balance is different between groups?

Yes: ANOVA requires that distribution of balance be normally distributed within each group. If not, you could transform your variable (log transfom?), or apply some other test (proc genmod?).

If you remove some observations (0-s or outliers) your results (inference) will be valid only to a subpopulations. (For example you simply want to exclude accounts with 0 or very high balance, because they do not represent the “real wealth of the account owners”)

Or you need to have a very good explanation why you are removing them. (For example if 0 is a data error.)

Re: Normality Test

Babloo — Wed, 04 Feb 2015 12:19:16 GMT

I need to determine whether mean balance is different between groups.

I agree with your point on 'If you remove some observations (0-s or outliers) your results (inference) will be valid only to a subpopulations. (For example you simply want to exclude accounts with 0 or very high balance, because they do not represent the “real wealth of the account owners”)'.

Now I wondering what values should be replaced for negative and zero values for my variable 'balance' before applying log normal distribution? Because 25% of my data is less than or equal to zero.

Re: Normality Test

SteveDenham — Wed, 04 Feb 2015 12:22:53 GMT

This really looks to me like an example where a nonparametric approach will be useful. There are zeroes and negative values and truly huge values in your data. Rather than looking at whether the means are different, consider whether the medians are different. A Mann-Whitney test, using PROC NPAR1WAY provides a distribution-free approach to this test.

Steve Denham

Re: Normality Test

Babloo — Wed, 04 Feb 2015 13:39:41 GMT

Can we make the inference in non-parametric test like we do in parametric test? e.g. I can test 'model balance=income age' in ANOVA or GLM. Since my data is non-normal how can I do the similar test in

non-parametric approach?

Re: Normality Test

SteveDenham — Wed, 04 Feb 2015 14:00:27 GMT

Time to read the documentation for PROC NPAR1WAY, paying particular attention to the examples there.

Steve Denham

Re: Normality Test

Babloo — Wed, 04 Feb 2015 14:43:18 GMT

Thanks,will have a look at documentation for PROC NPAR1WAY.

With regards to normality, I got a bell curve alongside mean=median and skewness (0.2) is also similar to Kurtosis (-1.08). but 'P' value is significant. e.g. p<0.001

So can we assume that our data is normal for the scenario as I mentioned above? or still we need to make it normal. I need to do ANOVA or GLM with that data.

Thanks!

Re: Normality Test

SteveDenham — Wed, 04 Feb 2015 15:17:41 GMT

RE:the statement:

With regards to normality, I got a bell curve alongside mean=median and skewness (0.2) is also similar to Kurtosis (-1.08). but 'P' value is significant. e.g. p<0.001

This is quite different from what was given above

( With the actual data mean is 5055 and the median is 68. Similarly skewness and kutosis is 8.5 and 166 respectively.)

So I assume this is after some sort of transform?

It still appears that the data are significantly different from being normally distributed, but that is not necessarily a stopping point. Please state how you got such different results this time.

Also, it is not necessary that the raw data be distributed normally to meet the assumptions of analysis of variance. The assumption is that the errors (residuals) be normally distributed. This can be checked by fitting the model of interest, getting the residuals in an output dataset, and then checking them for normality. For a Shapiro-Wilks test of normality, I would only reject the null hypothesis (of a normal distribution) if the P value were less than 0.001.

But much better than testing for normality would be looking at a QQ plot of the residuals. If those basically fit the diagonal without anything unusual, I would trust that the data were such that the assumption is nearly met, and depend on the robustness of the method.

Now if you get some extreme bends anywhere in the QQ plot, the nonparametric approach is probably more powerful than standard ANOVA.

Steve Denham

Re: Normality Test

Babloo — Thu, 05 Feb 2015 08:36:46 GMT

With the actual raw data mean is 5055 and the median is 68. Similarly skewness and kurtosis is 8.5 and 166 respectively followed by I did binning (15 bins) and removed the outliers. Now the mean is 14006.9 median is 14608.5 and skewness and kurtosis is 0.0013 and -1.2806 respectively. Considerably record count is reduced from 24000 to 600 where as p<0.001. Got a bell curve as well. Please suggest my final samples (600) is reasonably normal?

Then I replaced the actual value of balance1 (or amount) to log value (via balance1=log(balance1) ) and then I did a log normal and it produced a bell curve alongside mean=median and skewness (0.2) is also similar to Kurtosis (-1.08).

According to your advice, I tried to plot a Q-Q plot to check the normality with the code below. Attached the plot as well. Sounds it is non-normal as most of the data origin towards left. But I'm not very sure.

ods graphics on;

options maxmemquery=6M;

proc univariate data=normality_new;

QQPLOT balance1 / lognormal (sigma=2); /*I don't know about sigma value here*/

run;

ods graphics off;

So may I request you to view my plot and share your thoughts (or possibly verdict)?

Thanks.

Re: Normality Test

Reeza — Thu, 05 Feb 2015 10:02:48 GMT

Babloo wrote:

Considerably record count is reduced from 24000 to 600 where as p<0.001.

That would worry me. It seems like you've thrown away most of your data to obtain normality which means any results are probably not representative of your true business any more.

Re: Normality Test

Babloo — Thu, 05 Feb 2015 10:23:52 GMT

Then what should be the correct method?

When I did non-parametric method with the code below, it takes hours to complete. After referring the documents, I understood that it works well for limited samples (say less than 1000) whereas my data has 25000 observations.

proc npar1way wilcoxon data=want;

class income; /* categorized into three levels*/

var balance1;

exact;

run;

Any thoughts to re-direct me to obtain a normality or best fitted non-parametric procs?

Thanks.

Re: Normality Test

gergely_batho — Thu, 05 Feb 2015 10:32:41 GMT

I think you don't need the "exact" statement. With so many observations, your result will be "exact enough" with a simple (asymptotic) Kruskal-Wallis Test.

exact statement triggers a very expensive algorithm, it will take forever with 25000 obs.

Re: Normality Test

Babloo — Thu, 05 Feb 2015 10:52:49 GMT

Many thanks for your reply.

P value is <0.0001 with the below code

proc npar1way wilcoxon data=want;

class new_age;

var balance;

run;

Do we need to interpret only the P value from the output or anyother?

Before I close this thread, I would like to clarify below.

1. From the documentation, there is no separate keyword for Kruskal-Wallis test. Am I right?

WILCOXON

requests an analysis of Wilcoxon scores. When there are two classification levels (samples), this option produces the Wilcoxon rank-sum test. For any number of classification levels, this option produces the Kruskal-Wallis test. See the section Wilcoxon Scores for more information.

2. In case if I wish to include 2 classification variables (age,income) in this test, which proc should I use?

3. With regards to normal and log-normal distribution, whether is any difference whilst interpreting the outputs? Because we're modifying the actual value to a log value of a independent variable.

Thanks again for your inputs!