Contributor
Posts: 58

# Can Kappa Statistics account for Within Subject Correlation?

Hi All,

I have a study sample where I am trying to test the agreement between fainting episodes reported by patient self report vs. episodes reported by their spouse. I used Kappa to evaluate the agreement. However this is a repeated measures person-month data set where subjects are evaluated every month. Is there any way to account for within subject correlation for agreement testing? The sample data set and analytic code is listed below. Any help is much appreciated!

data have;
input Idnum Month Self_report_fainting_episodes Spouse_report_fainting_episodes;
datalines;
1    1    0    0
1    2    1    1
1    3    1    2
1    4    1    1
1    5    0    0
1    6    1    1
2    1    2    1
2    2    1    1
2    3    1    1
2    4    0    0
2    5    0    1
2    6    0    0
;
run;

proc freq data=have;
tables Self_report_fainting_episodes*Spouse_report_fainting_episodes / agree norow nocol nopercent;
test kappa;
run;

Posts: 2,655

## Re: Can Kappa Statistics account for Within Subject Correlation?

I hope we get some good ideas on this.  I was thinking of a joint model of count data, with possibly different Poisson distributions for spouse and self.  If we made this into a long dataset, something like:

data want;

length dist \$7;

set have;

source='Self  ';

response=self_reporting_fainting_episodes;

output;

source='Spouse';

response=spouse_reporting_fainting_episodes;

output;

keep idnum month source response ;

run;

class idnum month source;

model response=month source source*month /noint s dist=poi corrb;

random intercept month/subject=idnum type=ar(1); /* Might be many other possible covariance structures */

run;

The negative binomial would also be a candidate distribution.  This would give F tests of the differences between the two sources, conditional on the random effects.  Note that this is not 'agreement' as a kappa might measure, but rather (for the interaction) is the difference between the sources the same at all time points or not, and (for the main effect of source) if the difference at all time points is relatively the same (i.e., the interaction is not significant), what is the difference between the marginal means of the sources over all time points.

Steve Denham

Contributor
Posts: 58

## Re: Can Kappa Statistics account for Within Subject Correlation?

Thank you for your feedback Steve.

Based on my understanding (possibly naive), the issue with this approach is that it does not measure statistics pertaining to percent agreement or chance agreement. The statistics of interest is to test perfect agreement and partial agreement for which I generated a frequency table for the ordinal variables and used Kappa and Weighted Kappa as measures of reliability. However there is no weighting or adjusting for within subject variability in this approach.

Table of Self_report_fainting_episodes by Spouse_report_fainting_episodes
Self_report_fainting_episodesSpouse_report_fainting_episodes
012Total
0
 4
 1
 0
 5
1
 0
 5
 1
 6
2
 0
 1
 0
 1
Total
 4
 7
 1
 12

Although the interaction term in your model gives the difference between spouse and self report episodes at all time points, it is only the difference of marginal means with no measure of actual agreement.

Posts: 5,057

## Re: Can Kappa Statistics account for Within Subject Correlation?

You could start with a simplification of agreement classes. You could distinguish 4 cases :

1) both report no episodes (0, 0),

2) both report the same non-zero number of episodes (1, 1),

3) Spouse reports more episodes than Self (1, 0) and

4) Self reports more episodes than Spouse (0, 1)

The code would look like:

data have;
input Idnum Month Self_count Spouse_count;
datalines;
1    1    0    0
1    2    1    1
1    3    1    2
1    4    1    1
1    5    0    0
1    6    1    1
2    1    2    1
2    2    1    1
2    3    1    1
2    4    0    0
2    5    0    1
2    6    0    0
;

data want;
set have;
Self_cat = Self_count > 0 and Self_count >= Spouse_count;
Spouse_cat = Spouse_count > 0 and Spouse_count >= Self_count;
run;

proc freq data=want;
tables Self_cat*Spouse_cat / agree norow nocol nopercent plots=none;
run;

McNemar's test which is equivalent to Bowker's test for 2x2 tables would test for symmetry (i.e. both directions of disagreement are as frequent) and Kappa would measure agreement beyond chance (sort of).

Hth

PG

PG
Contributor
Posts: 58

## Re: Can Kappa Statistics account for Within Subject Correlation?

Dear PG,

Thank you for your feedback. The option you listed above reduces the n x n cells to a 2 x 2 table. However, that is not my concern. It is possible to get a weighted Kappa from ordinal data and make adjustments accordingly. I am concerned that (1) A person month data will have a lot of 0,0 cells and inflate the kappa statistics and (2) The degree of agreement may differ at person level so is there any possibility to adjust for within subject correlation? The other option was to simply the agreement classes by creating one individual level data set with the counts of number of episodes per person both by self and spouse report.

There is however one caveat to that. Consider this case:

data have;
input Idnum Month Self_count Spouse_count;
datalines;
3    1    0    1
3    2    1    0
3    3    1    1
3    4    1    1
3    5    0    1
3    6    1    0

;

run;

This can be reduced to one case as shown below:

idnum self spouse

3  4  4

Both spouse and self report number of episodes is the same non-zero number 4 i.e, (1, 1) in the 2 x 2 table cell. However, if you look more closely there is no agreement at month 1, 2, 5 and 6. That is the reason I set this up as a person month data set and not as counts per person.

Not sure if there is a way to get around this. Suggestions?

Posts: 2,655

## Re: Can Kappa Statistics account for Within Subject Correlation?

@pronabesh: Your example is why I was suggesting a linear models approach.  My reasoning was that if the two groups were significantly different, that could be detected through tests about source and source by month interaction.  With clever parameterization, we could even get measures of correlation between the fixed effect estimates that can be interpreted as a measure of agreement/concordance.

Steve Denham

Contributor
Posts: 58

## Re: Can Kappa Statistics account for Within Subject Correlation?

Steve,

A linear model approach could work. The investigator however is looking for test statistics for the measures of agreement and agreement by chance which in some ways is summarized by Kappa. I don't have the depth of knowledge to parametrize these measures within a linear model framework. If you have any suggestion, that will be very helpful.