topic Re: Statistical test in SAS Programming

Statistical test

newtriks — Tue, 24 Sep 2024 19:00:49 GMT

Is there a test to determine whether a column of individual student ratings (Likert 1-5) is significantly different from another column by the same rater of the same students at a later time?

Re: Statistical test

ballardw — Tue, 24 Sep 2024 19:08:33 GMT

Short answer: Yes.

Details would depend on what you want to test. Change per student(would require variable identifying the student)? Maybe a paired t-test of the mean difference.

Distribution of scores? Chi-square perhaps.

Can you state the research question the statistic is supposed to apply to?

Re: Statistical test

newtriks — Tue, 24 Sep 2024 19:12:27 GMT

The context is the evaluation of candidates for admission. Ratings are 1-5. We're trying to determine if a new element of the selection process matters so we're asking for ratings of each student before the rater sees this new element, and then after, to see if the new element has a significant effect on the ratings.

Thanks.

Re: Statistical test

PaigeMiller — Tue, 24 Sep 2024 19:18:40 GMT

@newtriks wrote:

The context is the evaluation of candidates for admission. Ratings are 1-5. We're trying to determine if a new element of the selection process matters so we're asking for ratings of each student before the rater sees this new element, and then after, to see if the new element has a significant effect on the ratings.

Are you talking about MEAN differences before/after? Or distribution of results before vs after? Or standard deviations? Or something else?

Re: Statistical test

newtriks — Tue, 24 Sep 2024 19:22:31 GMT

Not mean because we've nothing to compare the change to, other than "not change", which is our null. We're looking for ANY change. Looking at both the absolute value of the change AND whether there was a change regardless of size would both be of interest. Basically, are these two columns of numbers different in a significant way?

Re: Statistical test

PaigeMiller — Tue, 24 Sep 2024 19:26:26 GMT

I find this confusing. Why can't we subtract before minus after and compute the mean of the differences and see if it is statistically different from zero? Actually, the real question is more the first part: "Why can't we subtract before minus after and compute the mean of the differences"?

Re: Statistical test

newtriks — Tue, 24 Sep 2024 19:30:20 GMT

Wouldn't the "pluses" (say, a 1 to a 3) and the "minuses" (ie 4 to a 2) mitigate the effect when they're averaged with one another? We're looking for change, whatever the direction. If I'm understanding your approach, there could be a lot of individual change but potentially 0 in the aggregate if they all even out, no?

Re: Statistical test

PaigeMiller — Tue, 24 Sep 2024 19:38:20 GMT

Ok, that makes sense.

We could compare the distribution of before to the distribution of after. Does that sound like it meets your needs?

Re: Statistical test

newtriks — Tue, 24 Sep 2024 19:39:14 GMT

Yes.

Re: Statistical test

PaigeMiller — Tue, 24 Sep 2024 19:44:36 GMT

Lots of ways to do this in SAS

https://communities.sas.com/t5/Statistical-Procedures/formally-test-difference-of-two-distributions/td-p/740356

One of the examples appears to be ratings on a 1-5 scale.

Re: Statistical test

newtriks — Tue, 24 Sep 2024 19:42:34 GMT

Thanks!

Re: Statistical test

ballardw — Tue, 24 Sep 2024 19:59:51 GMT

You haven't stated what you want to test. You've described what the data contains, sort of.

Test would "is the mean score different(greater/less) between two groups" "is the distribution of scores the same/different" "is number of scores that increased/decreased significant" . Something about what you want to measure.

Since you seem to have before/after measures then some sort of paired (i.e. difference of individuals) scores is likely where you want to be.

Some possibilities: Sign test: which looks at the number of increases vs decreases (NOT magnitude just direction) and basically ignores ties. If there are "enough" Plus then scores increase, if enough minus then decreased if not enough of either then no statistical significance. One nice thing about this test is that it is appropriate for small samples.

Sign-rank test, ranks the differences and considers sign

Both of these can be done in Proc Univariate by calculating that difference (after-before per student).

If you have multiple "raters" that you want considered then sort the data by the "rater" variable and use that on a BY statement.

If you want to test a difference of the means then one of the basic would be a Ttest (Proc Ttest) and since your data is before after then likely PAIRED statement with Before variable * After Variable.

Appropriate tests also depend on sample sizes.

Re: Statistical test

Ksharp — Wed, 25 Sep 2024 01:42:00 GMT

You are doing Paried Test.

1) as ballardw said try Paried-TTest by proc ttest:

proc ttest data=have;
paired before*after;
run;

2) try Mixed or GEE model to take Before-After (Paired) Data as a repeated measures test.

https://support.sas.com/kb/46/997.html

Suppose you have data:

id  before  after
1   1            2
2    3            5
3    2            7
4    3            1
..............

And change it into longtidual data:

id  member response
1   before     1
1   after        2
2    before     3
2    after        5
.........

And using GEE model

 proc genmod data=indiv;
       class id member;
       model response = member;
       repeated subject=id;
       lsmeans member / diff cl;
       run;

Or Mixed model

 proc mixed data=indiv;
       class id member;
       model response = member /solution ;
       repeated  /subject=id type=ar(1);
       lsmeans member / diff cl;
       run;

Re: Statistical test

ChanceTGardener — Wed, 25 Sep 2024 16:59:13 GMT

One approach is to binarize the data and use McNemar's test in PROC FREQ.

For each student:

- if their Likert score increased, change their first observation to 0 and their second observation to 1

- if their Likert score decreased, change their first observation to 1 and their second observation to 0

- if their Likert score remained unchanged, code both observations as 0

McNemar's test determines if the discordant proportions (total 1's to 0's and total 0's to 1's across your sample) are significantly different from each other.

One caveat is you lose out on magnitude changes, because binarizing the data treats moving from 1 to 5 the same as moving from 3 to 4. If you care about magnitude changes, don't use this approach. This approach captures whether the aggregate change in perception (either positive or negative) is significantly different from the null hypothesis, which assumes no change in perception.

data in;
 input uid $ before after;
datalines;
A 0 0
B 0 1
C 0 1
D 0 0
E 1 0
F 0 1
G 0 1
H 0 0
I 0 1
J 0 1
K 0 1
L 0 0
M 0 0
N 1 0
O 0 1
P 0 1
Q 0 1
R 0 0
S 0 0
T 0 1
U 0 1
V 0 1
W 0 1
X 1 0
Y 0 1
Z 1 0
;

proc freq data=in; 
 tables before*after /agree nocol norow;  
run;

Re: Statistical test

newtriks — Thu, 26 Sep 2024 20:51:44 GMT

My interest is "did enough opinions change - whether up or down, doesn't matter - to be able to say that the new information changed opinions?" In other words, both increase and decrease should add to the significance of my point estimate, not negate each other until one comes out the winner. Can I work with McNemar in this way, or is there another test which does this?

Re: Statistical test

ChanceTGardener — Thu, 26 Sep 2024 21:18:10 GMT

McNemar's test tests whether the discordant proportions are significantly different from each other.

In the context of your problem, let:

P[UpVote] = sum of all students who upvoted

P[DownVote] = sum of all students who downvoted

Students whose scores do not change are not factored into the chi-square statistic.

The null hypothesis is P[UpVote] = P[DownVote], meaning the total upvotes and downvotes are statistically equivalent

The alternative is P[UpVote] != P[DownVote].

McNemar's test is testing whether or not these values are significantly different from each other.

Your description " "did enough opinions change - whether up or down, doesn't matter - to be able to say that the new information changed opinions?", does not make a distinction between upvotes and downvotes.

If you're treating upvotes and downvotes the same, then no, McNemar's test is not the right choice.

Re: Statistical test

newtriks — Thu, 26 Sep 2024 21:20:41 GMT

The null is "the votes did not change": every relative vote (before vs. after) = 0

Alternative is "the votes changed": every relative vote (or a significant enough amount of them) = 1

Re: Statistical test

newtriks — Thu, 26 Sep 2024 21:21:37 GMT

The null is "the votes did not change": every relative vote (before vs. after) = 0

CORRECTION Alternative is "the votes changed": ABSOLUTE VALUE of every relative vote (or a significant enough amount of them) = 1

Re: Statistical test

ChanceTGardener — Thu, 26 Sep 2024 21:45:28 GMT

For each student, let:

D = After - Before

Here, D=0 corresponds to no change in score.

If D is normally distributed, use the t-test mentioned by others earlier in this thread. If it is not normally distributed, use the Wilcoxon Signed-Rank Test, which does not assume normality. Both tests assume the distribution is symmetric.

This blog describes both in more detail and how to apply in SAS: https://blogs.sas.com/content/iml/2023/07/19/wilcoxon-signed-rank.html

Re: Statistical test

newtriks — Fri, 27 Sep 2024 19:17:49 GMT

So there is no way to measure absolute change? Theoretically, by the t-test every score could change by 2 but as long as half of them are +2 and half are -2, the test will reflect no change at all. Is this correct, or am I missing something?