topic Re: Proc univariate normality test and Shapiro-Wilk test in SAS Programming

Proc univariate normality test and Shapiro-Wilk test

LzEr23 — Sun, 23 Aug 2020 12:23:28 GMT

Dear all

I am having trouble with obtaining a normality test result using the Shapiro-Wilk (SW) test.

Usually, I have used the Univariate procedure with normal or normaltest options and was able to easily get normality test results for all four tests.

e.g.)

Proc univariate data=work.have normal;

var series;

run;

Now, I'm dealing with a larger dataset of which the observations are apparently greater than 2000, and the SAS suddenly does not show SW test results.

All other test statistics are fine, but only the SW is omitted.

I read that the SAS program automatically omits the SW test when the sample size is greater than 2000.

But, I still want to see the results using the SW test.

So, my question is, is there any other way that I can specify the SW method to test normality of the distribution of my dataset, which contains more than 2000 observations.

My dataset is shown as below.

data work.have; input date: yymmddn8. series; format date yymmddn8.; datalines; 20170101 501; 20170102 500.5; 20170103 505; ... 20171231 512; run;

I'd really appreciate if anyone could help me get this test result.

Hope my description was sufficient to understand.

But, it hadn't been so, please leave a note and ask.

Thanks in advance.

Re: Proc univariate normality test and Shapiro-Wilk test

PaigeMiller — Sun, 23 Aug 2020 12:58:16 GMT

The documentation is pretty clear

Shapiro-Wilk Statistic

If the sample size is less than or equal to 2000 and you specify the NORMAL option, PROC UNIVARIATE computes the Shapiro-Wilk statistic, W (also denoted as to emphasize its dependence on the sample size n).

So you can't get this statistic calculated for sample sizes above 2000. Other tests of normality should be used with sample sizes above 2000.

Re: Proc univariate normality test and Shapiro-Wilk test

ballardw — Sun, 23 Aug 2020 13:01:16 GMT

If you read this link https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test

you will find that some folks have extended the test to 5000 records but apparently that has not been added to SAS yet. I suspect a big issue is determining the critical points of the test statistic. The phrase from that link

The cutoff values for the statistics are calculated through Monte-Carlo simulations.

is an indicator to me that it is NOT a trivial exercise to expand sample sizes.

You do not indicate how big your sample might be. If it is greater than 5000 it looks like you would get your name attached if your work this out.

Short answer: Ain't gonna happen in SAS Proc Univariate.

Re: Proc univariate normality test and Shapiro-Wilk test

Ksharp — Mon, 24 Aug 2020 11:54:18 GMT

Maybe @Rick_SAS might write a blog (iml code) for this question ?