Statistical test

[newtriks]

Is there a test to determine whether a column of individual student ratings (Likert 1-5) is significantly different from another column by the same rater of the same students at a later time?

[ballardw]

Short answer: Yes.

Details would depend on what you want to test. Change per student(would require variable identifying the student)? Maybe a paired t-test of the mean difference.

Distribution of scores? Chi-square perhaps.

Can you state the research question the statistic is supposed to apply to?

[newtriks]

The context is the evaluation of candidates for admission. Ratings are 1-5. We're trying to determine if a new element of the selection process matters so we're asking for ratings of each student before the rater sees this new element, and then after, to see if the new element has a significant effect on the ratings.

Thanks.

[PaigeMiller]

---

@newtriks wrote:

The context is the evaluation of candidates for admission. Ratings are 1-5. We're trying to determine if a new element of the selection process matters so we're asking for ratings of each student before the rater sees this new element, and then after, to see if the new element has a significant effect on the ratings.

---

Are you talking about MEAN differences before/after? Or distribution of results before vs after? Or standard deviations? Or something else?

[newtriks]

Not mean because we've nothing to compare the change to, other than "not change", which is our null. We're looking for ANY change. Looking at both the absolute value of the change AND whether there was a change regardless of size would both be of interest. Basically, are these two columns of numbers different in a significant way?

[PaigeMiller]

I find this confusing. Why can't we subtract before minus after and compute the mean of the differences and see if it is statistically different from zero? Actually, the real question is more the first part: "Why can't we subtract before minus after and compute the mean of the differences"?

[newtriks]

Wouldn't the "pluses" (say, a 1 to a 3) and the "minuses" (ie 4 to a 2) mitigate the effect when they're averaged with one another? We're looking for change, whatever the direction. If I'm understanding your approach, there could be a lot of individual change but potentially 0 in the aggregate if they all even out, no?

[PaigeMiller]

Ok, that makes sense.

We could compare the distribution of before to the distribution of after. Does that sound like it meets your needs?

[newtriks]

Yes.

[PaigeMiller]

Lots of ways to do this in SAS

https://communities.sas.com/t5/Statistical-Procedures/formally-test-difference-of-two-distributions/...

One of the examples appears to be ratings on a 1-5 scale.

[newtriks]

Thanks!

[ballardw]

You haven't stated what you want to test. You've described what the data contains, sort of.

Test would "is the mean score different(greater/less) between two groups" "is the distribution of scores the same/different" "is number of scores that increased/decreased significant" . Something about what you want to measure.

Since you seem to have before/after measures then some sort of paired (i.e. difference of individuals) scores is likely where you want to be.

Some possibilities: Sign test: which looks at the number of increases vs decreases (NOT magnitude just direction) and basically ignores ties. If there are "enough" Plus then scores increase, if  enough minus then decreased if not enough of either then no statistical significance. One nice thing about this test is that it is appropriate for small samples.

Sign-rank test, ranks the differences and considers sign

Both of these can be done in Proc Univariate by calculating that difference (after-before per student).

If you have multiple "raters" that you want considered then sort the data by the "rater" variable and use that on a BY statement.

If you want to test a difference of the means then one of the basic would be a Ttest (Proc Ttest) and since your data is before after then likely PAIRED statement with Before variable * After Variable.

Appropriate tests also depend on sample sizes.

[Ksharp]

You are doing Paried Test.

1) as ballardw said try Paried-TTest by proc ttest:

proc ttest data=have;
paired before*after;
run;

2) try Mixed or GEE model to take Before-After (Paired) Data as a repeated measures test.

https://support.sas.com/kb/46/997.html

Suppose you have data:

id  before  after
1   1            2
2    3            5
3    2            7
4    3            1
..............

And change it into longtidual data:

id  member response
1   before     1
1   after        2
2    before     3
2    after        5
.........

And using GEE model 

 proc genmod data=indiv;
       class id member;
       model response = member;
       repeated subject=id;
       lsmeans member / diff cl;
       run;

Or Mixed model

 proc mixed data=indiv;
       class id member;
       model response = member /solution ;
       repeated  /subject=id type=ar(1);
       lsmeans member / diff cl;
       run;

[ChanceTGardener]

One approach is to binarize the data and use McNemar's test in PROC FREQ. 

For each student:

  - if their Likert score increased, change their first observation to 0 and their second observation to 1

  - if their Likert score decreased, change their first observation to 1 and their second observation to 0 

  - if their Likert score remained unchanged, code both observations as 0

McNemar's test determines if the discordant proportions (total 1's to 0's and total 0's to 1's across your sample) are significantly different from each other. 

One caveat is you lose out on magnitude changes, because binarizing the data treats moving from 1 to 5 the same as moving from 3 to 4. If you care about magnitude changes, don't use this approach. This approach captures whether the aggregate change in perception (either positive or negative) is significantly different from the null hypothesis, which assumes no change in perception. 

data in;
 input uid $ before after;
datalines;
A 0 0
B 0 1
C 0 1
D 0 0
E 1 0
F 0 1
G 0 1
H 0 0
I 0 1
J 0 1
K 0 1
L 0 0
M 0 0
N 1 0
O 0 1
P 0 1
Q 0 1
R 0 0
S 0 0
T 0 1
U 0 1
V 0 1
W 0 1
X 1 0
Y 0 1
Z 1 0
;

proc freq data=in; 
 tables before*after /agree nocol norow;  
run;

[newtriks]

My interest is "did enough opinions change - whether up or down, doesn't matter - to be able to say that the new information changed opinions?" In other words, both increase and decrease should add to the significance of my point estimate, not negate each other until one comes out the winner. Can I work with McNemar in this way, or is there another test which does this?