08-18-2015 02:22 PM
If you're not familiar with the term, top box scoring is often used with customer satisfaction surveys, and means that anything less than a perfect score is considered a failure. eBay starting doing this a few years ago, and a local retailer I shop at started doing this recently as well. I tried searching online and the only advantage I can find is that it's supposedly easier to explain to the layperson. But I can't see how it would be any easier than a simple mean. And it seems like it would have some significant disadvantages -- for example, a customer who gives a 1 (out of 5) is not going to shop at that store anymore. A customer who gives a 4 will continue to shop there. But top box treats them both the same. So why are companies using this scoring technique?
08-18-2015 03:35 PM
One reply question: what exactly does a mean of a 1 to 5 scale tell you? In some respect nothing as the length of the intervals between the categories are essentially meaningless. Plus a mean of 3 (hypothetical but basically true in concept for any mean of these scales) could be that every response was a 3, evenly divided between 1 and 5, ore uniform across the whole scale. So interpretation of a mean score requires much explanation unless the numbers actually measure something such as price, length, minutes etc.
A second in inflation of ratings, especially by repeat customers. As you say a very dissatisfied customer may never return so you only get a single 1 response from him but may get many 5s from repeat customers. I have actually worked with a customer with a 1 to 5 rating scale and was required to report a mean. We had to show it to 2 decimals to see any difference between the items rated as over 90% of all responses were 5.
Also the top, or bottom, box percentage is essentially a binomial distribution (1= 5, 0=every thing else) and there are lots of tools for working with binomials whereas the underlying distribution of your generic 1 to 5 scale is likely unknown.
08-18-2015 03:36 PM
Great question for stats.stackexchange.com, and probably more suited for the Statistical Procedures area of the forum.
I have never heard it referred to as top box, but ironically, I've used this method for its simplicity. I don't think you're restricted to a single score - you might take the top 2 or 3 depending on your scale, its a re-coding exercise.
08-19-2015 08:28 AM
Following up on Ed's reply: Lots of tools for binomially distributed data, and pretty easy to explain to a client. Once you get to multinomial distributions, you almost have to start reporting comparisons as odds ratios, and the business world doesn't seem to train people on how to think about those in the sense that biomedical or epidemiological folks are trained.
Here is my personal experience with a 5 point scale, devised by a client: 0 = no effect seen, 1 = mild effect seen, 2 = moderate effect seen, 3 = severe effect seen, 4 = animal died. So when I tried to explain that the difference between a response of 0 and a response of 1 is nothing like the difference between 1 and 2, and the difference between 0 and 2 isn't anything at all like the difference between 2 and 4. So, they said "Well, let's take the maximum score for the animal while it was observed, and then take the log of that number. And if the maximum is a zero, add 0.25 before taking the log, but don't add that to any of the other scores." And then calculate confidence intervals on the transformed values, back-transform and see if the lower bound is greater than 50% of the control means.