Comparing two means, independent samples, non-normal distribution

[TomKari]

Hi, all

I'm helping a colleague to evaluate some data. It pertains to two methods of capturing and correcting medical notes concerning patients:

1. The physician dictates the notes to an automated voice-to-text program, receives the text version of the notes, and corrects them manually.

2. The physician dictates the notes to a recording device, and the recording is then transcribed to a text file by a human at a keyboard. Again, the text file is then corrected manually by the physician.

Out of these processes come the word count of the resulting text file, and the time in seconds required by the physician to correct the errors in the file. The time is divided by the word count to result in a "seconds per word" statistic. We wish to compare the mean statistics for the two methods.

Although it is the same physician dictating both sets of notes, I'm treating them as independent samples. I'd appreciate feedback on whether this decision is correct.

Using PROC UNIVARIATE indicates that the results from the manual process aren't distributed normally. Based on what I've read, the independent samples and lack of normality indicate that a Wilcoxon rank-sum test (PROC NPAR1WAY) is the correct one to indicate whether the mean correction times are significantly different.

If I'm making any errors in this process, I would appreciate advice about how to approach it.

Thanks,
   Tom

[PaigeMiller]

Although it is the same physician dictating both sets of notes, I'm treating them as independent samples. I'd appreciate feedback on whether this decision is correct.

These seem paired to me.

Using PROC UNIVARIATE indicates that the results from the manual process aren't distributed normally. Based on what I've read, the independent samples and lack of normality indicate that a Wilcoxon rank-sum test (PROC NPAR1WAY) is the correct one to indicate whether the mean correction times are significantly different.

This is one way to determine if the means are different. However if the data is really paired, then you want to look at the distribution of the differences between method 1 and method 2 to see if it is normally distributed. Even if the difference is not normally distributed, sometimes the Central Limit Theorem comes to the rescue if you have enough observations, and you can treat it as normal.

[Reeza]

IIRC the t-test is pretty robust to deviations from normal - as long as they have the same distributions. There was a twitter debate on this a few weeks ago with some really good references (Yes, I'm that much of a data geek).

I would probably also be looking at grouping the analysis by physician and/or transcriber to ensure that there is no big deviations depending on who's doing the coding versus the machine. That's more EDA though.

We found some interesting physician and recorder counts that you can ask me about if we ever meet....one of the more memorable analysis projects of my career. 

Fun project though 😄

[PGStats · Accepted solution]

The timings should be considered paired only if the physician is having the same notes transcribed by both methods. Otherwise there is no possible pairing.

The non-parametric comparison will protect you from the influence of extreme observations at almost no cost in terms of power. You should stratify the test by physician.

[PaigeMiller]

The stimulus is the same. The treatment differs. This is paired. 

[PGStats]

I don't get it @PaigeMiller. Maybe our understanding of the nature of the data differs. How would the following obs be paired?:

physician caseId method timing
John           1 manual    1.0
John           2 manual    1.1
John           3 manual    1.2
John           4 machine   0.9
John           5 machine   1.3
John           6 machine   1.0

[PaigeMiller]

Then I would say that @TomKari has not given us enough information to determine if it is paired or not.

This also fits the description he gave in his original post

physician caseId method timing
John           1 manual    1.0
John           2 manual    1.1
John           3 manual    1.2
John           1 machine   0.9
John           2 machine   1.3
John           3 machine   1.0

[TomKari]

Hi, @PaigeMiller,  @Reeza, @PGStats

Thank you all for your insightful comments. They are a great help for someone relatively inexperienced at this.

First of all, the details that I should have included.

1. All of the notes are being dictated by a single physician.

2. They are not duplicates; he dictated notes for some patients using the first method, and some notes using the second method.

3. We don't know who the transcriber is, and at this point have no way to find out.

Based on your comments, I believe that the observations are not paired. Apologies for leaving out the pertinent details.

Although Reeza indicates that a t-test is probably fine, I'm can't be sure if the distributions of the two samples are the same. I think I'll stick with the Wilcoxon rank-sum.

Thank you again for all of your help!

   Tom

[PaigeMiller]

---

@TomKari wrote:

Hi, @PaigeMiller,  @Reeza, @PGStats

Thank you all for your insightful comments. They are a great help for someone relatively inexperienced at this.

First of all, the details that I should have included.

1. All of the notes are being dictated by a single physician.

2. They are not duplicates; he dictated notes for some patients using the first method, and some notes using the second method.

3. We don't know who the transcriber is, and at this point have no way to find out.

Based on your comments, I believe that the observations are not paired. Apologies for leaving out the pertinent details.

Although Reeza indicates that a t-test is probably fine, I'm can't be sure if the distributions of the two samples are the same. I think I'll stick with the Wilcoxon rank-sum.

Thank you again for all of your help!

   Tom

---

Yes, now I agree the data is not paired. In the future, if possible to do, setting this experiment up as paired would be a better statistical approach. To do this, the doctor would have to dictate the notes for a patient twice, using both methods.