Onyx | Level 15

## Comparing two means, independent samples, non-normal distribution

Hi, all

I'm helping a colleague to evaluate some data. It pertains to two methods of capturing and correcting medical notes concerning patients:

1. The physician dictates the notes to an automated voice-to-text program, receives the text version of the notes, and corrects them manually.

2. The physician dictates the notes to a recording device, and the recording is then transcribed to a text file by a human at a keyboard. Again, the text file is then corrected manually by the physician.

Out of these processes come the word count of the resulting text file, and the time in seconds required by the physician to correct the errors in the file. The time is divided by the word count to result in a "seconds per word" statistic. We wish to compare the mean statistics for the two methods.

Although it is the same physician dictating both sets of notes, I'm treating them as independent samples. I'd appreciate feedback on whether this decision is correct.

Using PROC UNIVARIATE indicates that the results from the manual process aren't distributed normally. Based on what I've read, the independent samples and lack of normality indicate that a Wilcoxon rank-sum test (PROC NPAR1WAY) is the correct one to indicate whether the mean correction times are significantly different.

If I'm making any errors in this process, I would appreciate advice about how to approach it.

Thanks,
Tom

1 ACCEPTED SOLUTION

Accepted Solutions
Opal | Level 21

## Re: Comparing two means, independent samples, non-normal distribution

The timings should be considered paired only if the physician is having the same notes transcribed by both methods. Otherwise there is no possible pairing.

The non-parametric comparison will protect you from the influence of extreme observations at almost no cost in terms of power. You should stratify the test by physician.

PG
8 REPLIES 8
Diamond | Level 26

## Re: Comparing two means, independent samples, non-normal distribution

Although it is the same physician dictating both sets of notes, I'm treating them as independent samples. I'd appreciate feedback on whether this decision is correct.

These seem paired to me.

Using PROC UNIVARIATE indicates that the results from the manual process aren't distributed normally. Based on what I've read, the independent samples and lack of normality indicate that a Wilcoxon rank-sum test (PROC NPAR1WAY) is the correct one to indicate whether the mean correction times are significantly different.

This is one way to determine if the means are different. However if the data is really paired, then you want to look at the distribution of the differences between method 1 and method 2 to see if it is normally distributed. Even if the difference is not normally distributed, sometimes the Central Limit Theorem comes to the rescue if you have enough observations, and you can treat it as normal.

--
Paige Miller
Super User

## Re: Comparing two means, independent samples, non-normal distribution

IIRC the t-test is pretty robust to deviations from normal - as long as they have the same distributions. There was a twitter debate on this a few weeks ago with some really good references (Yes, I'm that much of a data geek).

I would probably also be looking at grouping the analysis by physician and/or transcriber to ensure that there is no big deviations depending on who's doing the coding versus the machine. That's more EDA though.

We found some interesting physician and recorder counts that you can ask me about if we ever meet....one of the more memorable analysis projects of my career.

Fun project though 😄

Opal | Level 21

## Re: Comparing two means, independent samples, non-normal distribution

The timings should be considered paired only if the physician is having the same notes transcribed by both methods. Otherwise there is no possible pairing.

The non-parametric comparison will protect you from the influence of extreme observations at almost no cost in terms of power. You should stratify the test by physician.

PG
Diamond | Level 26

## Re: Comparing two means, independent samples, non-normal distribution

The stimulus is the same. The treatment differs. This is paired.

--
Paige Miller
Opal | Level 21

## Re: Comparing two means, independent samples, non-normal distribution

I don't get it @PaigeMiller. Maybe our understanding of the nature of the data differs. How would the following obs be paired?:

```physician caseId method timing
John           1 manual    1.0
John           2 manual    1.1
John           3 manual    1.2
John           4 machine   0.9
John           5 machine   1.3
John           6 machine   1.0

```
PG
Diamond | Level 26

## Re: Comparing two means, independent samples, non-normal distribution

Then I would say that @TomKari has not given us enough information to determine if it is paired or not.

This also fits the description he gave in his original post

```physician caseId method timing
John           1 manual    1.0
John           2 manual    1.1
John           3 manual    1.2
John           1 machine   0.9
John           2 machine   1.3
John           3 machine   1.0```

--
Paige Miller
Onyx | Level 15

## Re: Comparing two means, independent samples, non-normal distribution

Thank you all for your insightful comments. They are a great help for someone relatively inexperienced at this.

First of all, the details that I should have included.

1. All of the notes are being dictated by a single physician.

2. They are not duplicates; he dictated notes for some patients using the first method, and some notes using the second method.

3. We don't know who the transcriber is, and at this point have no way to find out.

Based on your comments, I believe that the observations are not paired. Apologies for leaving out the pertinent details.

Although Reeza indicates that a t-test is probably fine, I'm can't be sure if the distributions of the two samples are the same. I think I'll stick with the Wilcoxon rank-sum.

Thank you again for all of your help!

Tom

Diamond | Level 26

## Re: Comparing two means, independent samples, non-normal distribution

@TomKari wrote:

Thank you all for your insightful comments. They are a great help for someone relatively inexperienced at this.

First of all, the details that I should have included.

1. All of the notes are being dictated by a single physician.

2. They are not duplicates; he dictated notes for some patients using the first method, and some notes using the second method.

3. We don't know who the transcriber is, and at this point have no way to find out.

Based on your comments, I believe that the observations are not paired. Apologies for leaving out the pertinent details.

Although Reeza indicates that a t-test is probably fine, I'm can't be sure if the distributions of the two samples are the same. I think I'll stick with the Wilcoxon rank-sum.

Thank you again for all of your help!

Tom

Yes, now I agree the data is not paired. In the future, if possible to do, setting this experiment up as paired would be a better statistical approach. To do this, the doctor would have to dictate the notes for a patient twice, using both methods.

--
Paige Miller
Discussion stats
• 8 replies
• 3333 views
• 7 likes
• 4 in conversation