topic Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis in Statistical Procedures https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675842#M32387 <TABLE><TBODY><TR><TD>Subcorp</TD><TD>_Filename</TD><TD>ID</TD><TD>Start</TD><TD>End</TD><TD>Text</TD><TD>Comment</TD><TD>Role</TD><TD>ParentID</TD><TD>doc_completeness</TD><TD>appraisal</TD><TD>engagement</TD><TD>graduation</TD><TD>attitude</TD><TD>judgement</TD><TD>affect</TD><TD>appreciation</TD><TD>authorial_evaluation</TD><TD>non_authorial_evaluation</TD><TD>dis_satisfaction</TD><TD>un_happiness</TD><TD>in_security</TD><TD>dis_inclination</TD><TD>antipathy_affection</TD><TD>misery_cheer</TD><TD>ennui_interest</TD><TD>dis_pleasure</TD><TD>dis_quiet</TD><TD>dis_trust</TD><TD>tenacity</TD><TD>capacity</TD><TD>normality</TD><TD>unclear</TD><TD>veracity</TD><TD>propriety</TD><TD>composition</TD><TD>social_valuation</TD><TD>reaction</TD><TD>quality</TD><TD>impact</TD><TD>balance</TD><TD>complexity</TD><TD>inscribed</TD><TD>invoked</TD><TD>ambiguous</TD><TD>positive_attitude</TD><TD>negative_attitude</TD><TD>mono_glossic</TD><TD>hetero_glossic</TD><TD>expand</TD><TD>contract</TD><TD>proclaim</TD><TD>disclaim</TD><TD>deny</TD><TD>counter</TD><TD>pronounce</TD><TD>justify</TD><TD>endorse</TD><TD>concur</TD><TD>concede</TD><TD>affirm</TD><TD>entertain</TD><TD>attribute</TD><TD>distance</TD><TD>acknowledge</TD><TD>force</TD><TD>focus</TD><TD>intensification</TD><TD>quantification</TD><TD>degree</TD><TD>vigour</TD><TD>extent</TD><TD>number</TD><TD>mass</TD><TD>distribution</TD><TD>proximity</TD><TD>space</TD><TD>time</TD><TD>distrib_space</TD><TD>distrib_time</TD><TD>downscale</TD><TD>upscale</TD><TD>infusing</TD><TD>isolating</TD><TD>soften</TD><TD>sharpen</TD></TR><TR><TD>Texts</TD><TD>test1.txt</TD><TD>1</TD><TD>566</TD><TD>576</TD><TD>challenges</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD></TR><TR><TD>Texts</TD><TD>test2.txt</TD><TD>2</TD><TD>317</TD><TD>327</TD><TD>privileged</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD></TR><TR><TD>Texts</TD><TD>test2.txt</TD><TD>1</TD><TD>566</TD><TD>576</TD><TD>challenges</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD></TR></TBODY></TABLE><P>&nbsp;</P><P>I've chosen a couple of sentences online for example. Now I've read them into a sas data as shown in the above. It seems I need to transform the table into another form according to the previous pdf, right? In this case, the test1 is coderA and the test2 is the coderB.</P> Tue, 11 Aug 2020 08:26:04 GMT janex 2020-08-11T08:26:04Z Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675550#M32356 <P>There are two raters to code a text differently as in annotatepic.pdf. The output are the test1.txt and test2.txt.&nbsp;</P><P>The two raters may annotate different strings of text and may use different taggers (the code scheme is the same, but can choose different levels and different sub-tags). These tags are shown in the first line.</P><P>I want to use Cohen's kappa to calculate the reliability of the two raters. I'm new to SAS, and I need the help urgently. I highly appreciate if anybody could help me with it.&nbsp;</P> Mon, 10 Aug 2020 10:18:14 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675550#M32356 janex 2020-08-10T10:18:14Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675551#M32357 <P>I read articles about the calculation of <EM>k</EM>, but I don't know how to deal with the data, how should I transform the data into a suitable one for cohen's kappa so as to compare the tokens of two different texts (two different raters)</P> Mon, 10 Aug 2020 10:22:12 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675551#M32357 janex 2020-08-10T10:22:12Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675557#M32358 <P>Welcome to the SAS Community!</P> <P>&nbsp;</P> <P>You mind find the answer to your question in this document:</P> <P>&nbsp;</P> <P><A href="https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/180-30.pdf" target="_blank">https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/180-30.pdf</A></P> Mon, 10 Aug 2020 11:40:20 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675557#M32358 Norman21 2020-08-10T11:40:20Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675775#M32384 <P>Thank you ^^&nbsp;</P><P>I read this article, but the question is I don't know how should I transform my data into an approapriate form to do that.</P> Mon, 10 Aug 2020 22:18:15 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675775#M32384 janex 2020-08-10T22:18:15Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675836#M32386 <P>I'm not sure how to interpret the test text, but it looks like you have to assign the data to each variable. However, the variable names will need to be changed. For example, "in/security" is not a valid SAS variable name; you might want to change this to "in_security". Here are a couple of examples of what needs to be done:</P> <P>&nbsp;</P> <P><A href="https://documentation.sas.com/?docsetId=lestmtsref&amp;docsetTarget=p0114gachtut3nn1and4ap8ke9nf.htm&amp;docsetVersion=3.1&amp;locale=en">https://documentation.sas.com/?docsetId=lestmtsref&amp;docsetTarget=p0114gachtut3nn1and4ap8ke9nf.htm&amp;docsetVersion=3.1&amp;locale=en</A></P> <P>&nbsp;</P> <P>FYI, I managed to find the meeting from where the transcription was made:</P> <P>&nbsp;</P> <P><A href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.c-span.org%2Fvideo%2F%3F327062-2%2Fpresident-obama-remarks-2015-white-house-conference-aging&amp;data=02%7C01%7C%7Cb6067f597dc649a7052108d83da09836%7C1faf88fea9984c5b93c9210a11d9a5c2%7C0%7C0%7C637327107251152843&amp;sdata=eN2Wxqdnmg%2FjHbc5lpyhGVt0wd5ISqhMmytoIH1s0R4%3D&amp;reserved=0" target="_blank">https://www.c-span.org/video/?327062-2/president-obama-remarks-2015-white-house-conference-</A></P> Tue, 11 Aug 2020 07:39:19 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675836#M32386 Norman21 2020-08-11T07:39:19Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675842#M32387 <TABLE><TBODY><TR><TD>Subcorp</TD><TD>_Filename</TD><TD>ID</TD><TD>Start</TD><TD>End</TD><TD>Text</TD><TD>Comment</TD><TD>Role</TD><TD>ParentID</TD><TD>doc_completeness</TD><TD>appraisal</TD><TD>engagement</TD><TD>graduation</TD><TD>attitude</TD><TD>judgement</TD><TD>affect</TD><TD>appreciation</TD><TD>authorial_evaluation</TD><TD>non_authorial_evaluation</TD><TD>dis_satisfaction</TD><TD>un_happiness</TD><TD>in_security</TD><TD>dis_inclination</TD><TD>antipathy_affection</TD><TD>misery_cheer</TD><TD>ennui_interest</TD><TD>dis_pleasure</TD><TD>dis_quiet</TD><TD>dis_trust</TD><TD>tenacity</TD><TD>capacity</TD><TD>normality</TD><TD>unclear</TD><TD>veracity</TD><TD>propriety</TD><TD>composition</TD><TD>social_valuation</TD><TD>reaction</TD><TD>quality</TD><TD>impact</TD><TD>balance</TD><TD>complexity</TD><TD>inscribed</TD><TD>invoked</TD><TD>ambiguous</TD><TD>positive_attitude</TD><TD>negative_attitude</TD><TD>mono_glossic</TD><TD>hetero_glossic</TD><TD>expand</TD><TD>contract</TD><TD>proclaim</TD><TD>disclaim</TD><TD>deny</TD><TD>counter</TD><TD>pronounce</TD><TD>justify</TD><TD>endorse</TD><TD>concur</TD><TD>concede</TD><TD>affirm</TD><TD>entertain</TD><TD>attribute</TD><TD>distance</TD><TD>acknowledge</TD><TD>force</TD><TD>focus</TD><TD>intensification</TD><TD>quantification</TD><TD>degree</TD><TD>vigour</TD><TD>extent</TD><TD>number</TD><TD>mass</TD><TD>distribution</TD><TD>proximity</TD><TD>space</TD><TD>time</TD><TD>distrib_space</TD><TD>distrib_time</TD><TD>downscale</TD><TD>upscale</TD><TD>infusing</TD><TD>isolating</TD><TD>soften</TD><TD>sharpen</TD></TR><TR><TD>Texts</TD><TD>test1.txt</TD><TD>1</TD><TD>566</TD><TD>576</TD><TD>challenges</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD></TR><TR><TD>Texts</TD><TD>test2.txt</TD><TD>2</TD><TD>317</TD><TD>327</TD><TD>privileged</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD></TR><TR><TD>Texts</TD><TD>test2.txt</TD><TD>1</TD><TD>566</TD><TD>576</TD><TD>challenges</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>&nbsp;</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>1</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD><TD>0</TD></TR></TBODY></TABLE><P>&nbsp;</P><P>I've chosen a couple of sentences online for example. Now I've read them into a sas data as shown in the above. It seems I need to transform the table into another form according to the previous pdf, right? In this case, the test1 is coderA and the test2 is the coderB.</P> Tue, 11 Aug 2020 08:26:04 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675842#M32387 janex 2020-08-11T08:26:04Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675870#M32388 <P>I just ran through all the steps and codes in&nbsp;<A href="https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/180-30.pdf" target="_blank">https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/180-30.pdf</A>&nbsp;. I kind of knew what it is now. The most important thing is to transform my data into the correct form for the calculation of kappa.</P> Tue, 11 Aug 2020 10:33:50 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675870#M32388 janex 2020-08-11T10:33:50Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675878#M32389 <P>You might find PROC TRANSPOSE useful for this task:</P> <P>&nbsp;</P> <P><A href="https://support.sas.com/resources/papers/proceedings09/060-2009.pdf" target="_blank">https://support.sas.com/resources/papers/proceedings09/060-2009.pdf</A></P> Tue, 11 Aug 2020 11:18:16 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/675878#M32389 Norman21 2020-08-11T11:18:16Z Re: Need help with inter-rater-reliability with cohen's kappa for discourse analysis https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/676027#M32399 <P>Thank you very much ^^&nbsp;</P><P>NZ got locked down again. I might try it next week.</P> Tue, 11 Aug 2020 23:06:00 GMT https://communities.sas.com/t5/Statistical-Procedures/Need-help-with-inter-rater-reliability-with-cohen-s-kappa-for/m-p/676027#M32399 janex 2020-08-11T23:06:00Z