Need help with inter-rater-reliability with cohen's kappa for discourse analysis

[janex]

There are two raters to code a text differently as in annotatepic.pdf. The output are the test1.txt and test2.txt. 

The two raters may annotate different strings of text and may use different taggers (the code scheme is the same, but can choose different levels and different sub-tags). These tags are shown in the first line.

I want to use Cohen's kappa to calculate the reliability of the two raters. I'm new to SAS, and I need the help urgently. I highly appreciate if anybody could help me with it. 

[janex]

I read articles about the calculation of k, but I don't know how to deal with the data, how should I transform the data into a suitable one for cohen's kappa so as to compare the tokens of two different texts (two different raters)

[Norman21]

Welcome to the SAS Community!

You mind find the answer to your question in this document:

https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/180-30.pdf

[janex]

Thank you ^^ 

I read this article, but the question is I don't know how should I transform my data into an approapriate form to do that.

[Norman21]

I'm not sure how to interpret the test text, but it looks like you have to assign the data to each variable. However, the variable names will need to be changed. For example, "in/security" is not a valid SAS variable name; you might want to change this to "in_security". Here are a couple of examples of what needs to be done:

https://documentation.sas.com/?docsetId=lestmtsref&docsetTarget=p0114gachtut3nn1and4ap8ke9nf.htm&doc...

FYI, I managed to find the meeting from where the transcription was made:

https://www.c-span.org/video/?327062-2/president-obama-remarks-2015-white-house-conference-

[janex]

Subcorp	_Filename	ID	Start	End	Text	Comment	Role	ParentID	doc_completeness	appraisal	engagement	graduation	attitude	judgement	affect	appreciation	authorial_evaluation	non_authorial_evaluation	dis_satisfaction	un_happiness	in_security	dis_inclination	antipathy_affection	misery_cheer	ennui_interest	dis_pleasure	dis_quiet	dis_trust	tenacity	capacity	normality	unclear	veracity	propriety	composition	social_valuation	reaction	quality	impact	balance	complexity	inscribed	invoked	ambiguous	positive_attitude	negative_attitude	mono_glossic	hetero_glossic	expand	contract	proclaim	disclaim	deny	counter	pronounce	justify	endorse	concur	concede	affirm	entertain	attribute	distance	acknowledge	force	focus	intensification	quantification	degree	vigour	extent	number	mass	distribution	proximity	space	time	distrib_space	distrib_time	downscale	upscale	infusing	isolating	soften	sharpen
Texts	test1.txt	1	566	576	challenges					1	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Texts	test2.txt	2	317	327	privileged					1	0	0	1	0	1	0	1	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Texts	test2.txt	1	566	576	challenges					1	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

I've chosen a couple of sentences online for example. Now I've read them into a sas data as shown in the above. It seems I need to transform the table into another form according to the previous pdf, right? In this case, the test1 is coderA and the test2 is the coderB.

[janex]

I just ran through all the steps and codes in https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/180-30.pdf . I kind of knew what it is now. The most important thing is to transform my data into the correct form for the calculation of kappa.

[Norman21]

You might find PROC TRANSPOSE useful for this task:

https://support.sas.com/resources/papers/proceedings09/060-2009.pdf

[janex]

Thank you very much ^^ 

NZ got locked down again. I might try it next week.