BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
Linlin
Lapis Lazuli | Level 10

Hi Experts,

We ask people to draw clocks for us. We used to score the clocks manually. there are about 100 data points (100 variables), most of variables are categorical (example: is the clock outline present? 0=No, 1=Yes, 2=outline is  not a circle); Now people use digital pens to draw the clocks so the data would be generated automatically. I need to find out how good the automatically generated data is by camparing it with the manually scored data (I have 300 observations). What should I do?

Any help is highly appreciated!

1 ACCEPTED SOLUTION

Accepted Solutions
PGStats
Opal | Level 21

I would start with something like this :

data manual;
input clock$ var1@@;
cards;
clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2
clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2
clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2
clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2
clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2
clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2
clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2
clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2
clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2
;

data digit;
input clock$ var1@@;
cards;
clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2
clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2
clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2
clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2
clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2
clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2
clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2
clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2
clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2
;

proc sql;
create table compare as
select m.clock, m.var1 as mVar1, d.var1 as dVar1
from manual as m inner join digit as d on m.clock=d.clock;
quit;

proc freq data=compare;
table mVar1*dVar1 / agree noprint plots=agreeplot;
test kappa;
run;

PG

PG

View solution in original post

13 REPLIES 13
Reeza
Super User

Do you have multiple people grading the clock drawing or is the person considered the 'correct' answer?

Some terms that might be helpful:

Inter-rater observer problem - Kappa statistics. 

Linlin
Lapis Lazuli | Level 10

Hi Reeza,

Thank you and congratulations!!!

Manually scored data is considered the "correct" answer even there are multiple people score the clock drawings.

Thanks - Linlin

art297
Opal | Level 21

: Congratulation from me too!

: Since our statistically oriented colleagues have mentioned reliability, I have to ask a question.  Has anyone ever measured the degree of reliability of the manual scoring method?  There are at least two issues that you should be concerned with:

validity: are both tests, or possibly even the collection of the 100 data points, accurately measuring that which they were designed to measure?

and

How consistently are they measuring whatever it is that they are measuring.

Both are important, but I was suggesting looking more at validity and, in the process, finding out why the two methods differ and provide decision makers with the necessary information to make a determination.

Linlin
Lapis Lazuli | Level 10

Thanks Art!

We wouldn’t change anything about the manual scoring(the
project started several decades ago) , but the Digit pen developer could change
his scoring. We have 300 clocks scored by both methods. Once we decide the
Digit scores are close enough to the manual scores we would stop the manual
scoring.

Reeza
Super User

Thanks LinLin/Art!  It is definitely nice to be in such nice company Smiley Wink

LinLin there's been a bit of work around this in terms of mammography and using machines to read tumour information, so a bit of literature search may turn up what other people are using to analyze this type of information.

art297
Opal | Level 21

I would start with:

1. A sample of around 50 of the manually scored drawings.

2. I would trace them, as accurately as possible, using the new method

3. I would identify which data points have discrepancies between the two methods (both in terms of mean response and variance)

4. Starting with those with the most discrepancies I would determine whether the automated ratings were correct and, if not, determine what is wrong with the algorithm

PGStats
Opal | Level 21

Look for agreement statistics (such as Cohen's Kappa) in proc freq. - PG

PG
Linlin
Lapis Lazuli | Level 10

Thank you Art and PG!

For one variable, could I do something like below?

data manual;

input clock$ var1@@;

cards;

clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2

clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2

clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2

clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2

clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2

clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2

clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2

clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2

clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2

;

proc sort;

by clock;

run;

data digit;

input clock$ var1@@;

cards;

clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2

clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2

clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2

clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2

clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2

clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2

clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2

clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2

clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2

;

proc sort;

by clock;

run;

data combined;

merge manual(in=a) digit(in=b rename=(var1=di));

by clock;

if a and b;

com1=ifn(var1=di,0,1);

proc print;run;

proc freq data=combined;

   tables com1/chisq;

run;

Thanks,

Linlin

PGStats
Opal | Level 21

I would start with something like this :

data manual;
input clock$ var1@@;
cards;
clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2
clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2
clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2
clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2
clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2
clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2
clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2
clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2
clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2
;

data digit;
input clock$ var1@@;
cards;
clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2
clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2
clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2
clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2
clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2
clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2
clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2
clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2
clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2
;

proc sql;
create table compare as
select m.clock, m.var1 as mVar1, d.var1 as dVar1
from manual as m inner join digit as d on m.clock=d.clock;
quit;

proc freq data=compare;
table mVar1*dVar1 / agree noprint plots=agreeplot;
test kappa;
run;

PG

PG
Linlin
Lapis Lazuli | Level 10

Thank you PG!

How should I interpret the output:smileysilly::smileyblush:? .

PGStats
Opal | Level 21

Kappa gives you an objective measure of agreement between methods. The idea would be to get the Kappa for each variable and identify where the greatest discrepancies are. If changes are proposed to the automatic scoring procedure you could also assess them by comparing new and previous Kappas. - PG

PG
Linlin
Lapis Lazuli | Level 10

Thank you PG! What is the criterion would you use to decide the discrepancy is small enough? How about the example you provided? is that small enough to be accepted?

PGStats
Opal | Level 21

Though questions!

One way to decide would be to look at the agreement between multiple manual evaluations of the same drawings, if you have some, and try to stay within that margin with the new method.

But it might be more appropriate (and simpler) to concentrate on the final outcome of these clock drawing exercises. It is up to the investigator to decide how much change he/she is willing to accept as a consequence of the change of method.

hth

PG

PG

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 13 replies
  • 2048 views
  • 7 likes
  • 4 in conversation