turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- What procedure should I use?

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 05:12 PM

Hi Experts,

We ask people to draw clocks for us. We used to score the clocks manually. there are about 100 data points (100 variables), most of variables are categorical (example: is the clock outline present? 0=No, 1=Yes, 2=outline is not a circle); Now people use digital pens to draw the clocks so the data would be generated automatically. I need to find out how good the automatically generated data is by camparing it with the manually scored data (I have 300 observations). What should I do?

Any help is highly appreciated!

Accepted Solutions

Solution

01-09-2013
09:09 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 09:09 PM

I would start with something like this :

**data manual;****input clock$ var1@@;****cards;****clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2 ****clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2 ****clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2 ****clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2 ****clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2 ****clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2 ****clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2 ****clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2 ****clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2 ****;**

**data digit;****input clock$ var1@@;****cards;****clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2 ****clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2 ****clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2 ****clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2 ****clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2 ****clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2 ****clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2 ****clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2 ****clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2 ****;**

** **

**proc sql;****create table compare as****select m.clock, m.var1 as mVar1, d.var1 as dVar1****from manual as m inner join digit as d on m.clock=d.clock;****quit;**

** **

**proc freq data=compare;****table mVar1*dVar1 / agree noprint plots=agreeplot;****test kappa;****run;**

PG

PG

All Replies

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 05:35 PM

Do you have multiple people grading the clock drawing or is the person considered the 'correct' answer?

Some terms that might be helpful:

Inter-rater observer problem - Kappa statistics.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 07:05 PM

Hi Reeza,

Thank you and congratulations!!!

Manually scored data is considered the "correct" answer even there are multiple people score the clock drawings.

Thanks - Linlin

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 11:48 PM

: Since our statistically oriented colleagues have mentioned reliability, I have to ask a question. Has anyone ever measured the degree of reliability of the manual scoring method? There are at least two issues that you should be concerned with:

validity: are both tests, or possibly even the collection of the 100 data points, accurately measuring that which they were designed to measure?

and

How consistently are they measuring whatever it is that they are measuring.

Both are important, but I was suggesting looking more at validity and, in the process, finding out why the two methods differ and provide decision makers with the necessary information to make a determination.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-10-2013 08:43 AM

Thanks Art!

We wouldn’t change anything about the manual scoring(the

project started several decades ago) , but the Digit pen developer could change

his scoring. We have 300 clocks scored by both methods. Once we decide the

Digit scores are close enough to the manual scores we would stop the manual

scoring.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-10-2013 10:57 AM

Thanks LinLin/Art! It is definitely nice to be in such nice company

LinLin there's been a bit of work around this in terms of mammography and using machines to read tumour information, so a bit of literature search may turn up what other people are using to analyze this type of information.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 05:36 PM

I would start with:

1. A sample of around 50 of the manually scored drawings.

2. I would trace them, as accurately as possible, using the new method

3. I would identify which data points have discrepancies between the two methods (both in terms of mean response and variance)

4. Starting with those with the most discrepancies I would determine whether the automated ratings were correct and, if not, determine what is wrong with the algorithm

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 05:37 PM

Look for agreement statistics (such as Cohen's Kappa) in proc freq. - PG

PG

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 08:32 PM

Thank you Art and PG!

For one variable, could I do something like below?

data manual;

input clock$ var1@@;

cards;

clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2

clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2

clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2

clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2

clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2

clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2

clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2

clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2

clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2

;

proc sort;

by clock;

run;

data digit;

input clock$ var1@@;

cards;

clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2

clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2

clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2

clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2

clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2

clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2

clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2

clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2

clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2

;

proc sort;

by clock;

run;

data combined;

merge manual(in=a) digit(in=b rename=(var1=di));

by clock;

if a and b;

com1=ifn(var1=di,0,1);

proc print;run;

proc freq data=combined;

tables com1/chisq;

run;

Thanks,

Linlin

Solution

01-09-2013
09:09 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 09:09 PM

I would start with something like this :

**data manual;****input clock$ var1@@;****cards;****clock01 0 clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2 ****clock11 0 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2 ****clock21 0 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2 ****clock31 0 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2 ****clock41 0 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2 ****clock51 0 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2 ****clock61 0 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2 ****clock71 0 clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2 ****clock81 0 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2 ****;**

**data digit;****input clock$ var1@@;****cards;****clock01 . clock02 1 clock03 2 clock04 0 clock05 1 clock06 2 clock07 0 clock08 1 clock09 2 ****clock11 1 clock12 1 clock13 2 clock14 0 clock15 1 clock16 2 clock17 0 clock18 1 clock19 2 ****clock21 2 clock22 1 clock23 2 clock24 0 clock25 1 clock26 2 clock27 0 clock28 1 clock29 2 ****clock31 1 clock32 1 clock33 2 clock34 0 clock35 1 clock36 2 clock37 0 clock38 1 clock39 2 ****clock41 2 clock42 1 clock43 2 clock44 0 clock45 1 clock46 2 clock47 0 clock48 1 clock49 2 ****clock51 2 clock52 1 clock53 2 clock54 0 clock55 1 clock56 2 clock57 0 clock58 1 clock59 2 ****clock61 1 clock62 1 clock63 2 clock64 0 clock65 1 clock66 2 clock67 0 clock68 1 clock69 2 ****clock71 . clock72 1 clock73 2 clock74 0 clock75 1 clock76 2 clock77 0 clock78 1 clock79 2 ****clock81 1 clock82 1 clock83 2 clock84 0 clock85 1 clock86 2 clock87 0 clock88 1 clock89 2 ****;**

** **

**proc sql;****create table compare as****select m.clock, m.var1 as mVar1, d.var1 as dVar1****from manual as m inner join digit as d on m.clock=d.clock;****quit;**

** **

**proc freq data=compare;****table mVar1*dVar1 / agree noprint plots=agreeplot;****test kappa;****run;**

PG

PG

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-09-2013 10:08 PM

Thank you PG!

How should I interpret the output:smileysilly::smileyblush:? .

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-10-2013 08:17 AM

Kappa gives you an objective measure of agreement between methods. The idea would be to get the Kappa for each variable and identify where the greatest discrepancies are. If changes are proposed to the automatic scoring procedure you could also assess them by comparing new and previous Kappas. - PG

PG

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-10-2013 08:23 AM

Thank you PG! What is the criterion would you use to decide the discrepancy is small enough? How about the example you provided? is that small enough to be accepted?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-10-2013 10:55 AM

Though questions!

One way to decide would be to look at the agreement between multiple manual evaluations of the same drawings, if you have some, and try to stay within that margin with the new method.

But it might be more appropriate (and simpler) to concentrate on the final outcome of these clock drawing exercises. It is up to the investigator to decide how much change he/she is willing to accept as a consequence of the change of method.

hth

PG

PG