What procedure should I use?

[Linlin]

Hi Experts,

We ask people to draw clocks for us. We used to score the clocks manually. there are about 100 data points (100 variables), most of variables are categorical (example: is the clock outline present? 0=No, 1=Yes, 2=outline is  not a circle); Now people use digital pens to draw the clocks so the data would be generated automatically. I need to find out how good the automatically generated data is by camparing it with the manually scored data (I have 300 observations). What should I do?

Any help is highly appreciated!

[Reeza]

Do you have multiple people grading the clock drawing or is the person considered the 'correct' answer?

Some terms that might be helpful:

Inter-rater observer problem - Kappa statistics. 

[Linlin]

Hi Reeza,

Thank you and congratulations!!!

Manually scored data is considered the "correct" answer even there are multiple people score the clock drawings.

Thanks - Linlin

[art297]

: Congratulation from me too!

: Since our statistically oriented colleagues have mentioned reliability, I have to ask a question.  Has anyone ever measured the degree of reliability of the manual scoring method?  There are at least two issues that you should be concerned with:

validity: are both tests, or possibly even the collection of the 100 data points, accurately measuring that which they were designed to measure?

and

How consistently are they measuring whatever it is that they are measuring.

Both are important, but I was suggesting looking more at validity and, in the process, finding out why the two methods differ and provide decision makers with the necessary information to make a determination.

[Linlin]

Thanks Art!

We wouldn’t change anything about the manual scoring(the
project started several decades ago) , but the Digit pen developer could change
his scoring. We have 300 clocks scored by both methods. Once we decide the
Digit scores are close enough to the manual scores we would stop the manual
scoring.

[Reeza]

Thanks LinLin/Art!  It is definitely nice to be in such nice company

LinLin there's been a bit of work around this in terms of mammography and using machines to read tumour information, so a bit of literature search may turn up what other people are using to analyze this type of information.

[art297]

I would start with:

1. A sample of around 50 of the manually scored drawings.

2. I would trace them, as accurately as possible, using the new method

3. I would identify which data points have discrepancies between the two methods (both in terms of mean response and variance)

4. Starting with those with the most discrepancies I would determine whether the automated ratings were correct and, if not, determine what is wrong with the algorithm

[PGStats]

Look for agreement statistics (such as Cohen's Kappa) in proc freq. - PG

[Linlin]

Thank you Art and PG!

For one variable, could I do something like below?

data manual;

input clock$ var1@@;

cards;

clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2

clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2

clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2

clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2

clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2

clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2

clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2

clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2

clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2

;

proc sort;

by clock;

run;

data digit;

input clock$ var1@@;

cards;

clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2

clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2

clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2

clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2

clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2

clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2

clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2

clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2

clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2

;

proc sort;

by clock;

run;

data combined;

merge manual(in=a) digit(in=b rename=(var1=di));

by clock;

if a and b;

com1=ifn(var1=di,0,1);

proc print;run;

proc freq data=combined;

   tables com1/chisq;

run;

Thanks,

Linlin

[PGStats · Accepted solution]

I would start with something like this :

data manual;
input clock$ var1@@;
cards;
clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2
clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2
clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2
clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2
clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2
clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2
clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2
clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2
clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2
;

data digit;
input clock$ var1@@;
cards;
clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2
clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2
clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2
clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2
clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2
clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2
clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2
clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2
clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2
;

proc sql;
create table compare as
select m.clock, m.var1 as mVar1, d.var1 as dVar1
from manual as m inner join digit as d on m.clock=d.clock;
quit;

proc freq data=compare;
table mVar1*dVar1 / agree noprint plots=agreeplot;
test kappa;
run;

PG

[Linlin]

Thank you PG!

How should I interpret the output:smileysilly::smileyblush:? .

[PGStats]

Kappa gives you an objective measure of agreement between methods. The idea would be to get the Kappa for each variable and identify where the greatest discrepancies are. If changes are proposed to the automatic scoring procedure you could also assess them by comparing new and previous Kappas. - PG

[Linlin]

Thank you PG! What is the criterion would you use to decide the discrepancy is small enough? How about the example you provided? is that small enough to be accepted?

[PGStats]

Though questions!

One way to decide would be to look at the agreement between multiple manual evaluations of the same drawings, if you have some, and try to stay within that margin with the new method.

But it might be more appropriate (and simpler) to concentrate on the final outcome of these clock drawing exercises. It is up to the investigator to decide how much change he/she is willing to accept as a consequence of the change of method.

hth

PG