topic Re: How root mean squared standard deviation (RMSSTD) is calculated for Text document clustering? in SAS Data Science

How root mean squared standard deviation (RMSSTD) is calculated for Text document clustering?

AbhishekVerma1985 — Sat, 26 Nov 2016 13:06:42 GMT

How root mean squared standard deviation (RMSSTD) is calculated for Text document clustering? There is no mathematics is given in any of SAS documentation or Help regarding this.

Re: How root mean squared standard deviation (RMSSTD) is calculated for Text document clustering?

RussAlbright — Wed, 07 Dec 2016 03:00:36 GMT

if K is the number of dimensions used in the clustering, m is the number of docs in the cluster, and err is the sum of the m*k squared errors, then it looks like it is calculated to be

rmstd = sqrt(err/((m-1)*K)), unless m = 1 and then the value is 0.

Re: How root mean squared standard deviation (RMSSTD) is calculated for Text document clustering?

AbhishekVerma1985 — Sat, 27 May 2017 06:01:52 GMT

Dear Russ,

It is little confusing to me. I am not able to understand "err is the sum of the m*k squared errors" it will be very helpful if you explain this.

Re: How root mean squared standard deviation (RMSSTD) is calculated for Text document clustering?

RussAlbright — Wed, 31 May 2017 09:48:38 GMT

Each document is a K dimensional vector.

Similarly, the mean of the cluster is a k dimensional vector where each component is an average of the corresponding component for each of the m documents.

A document error is the square root of the sum of the squared differences of each of its k components with each of the k components of the mean of the cluster.

The RMSSTD is a an error for the entire cluster so to incorporate all documents from the cluster in this err caculation, it becomes the sum of the squared differences for every component of every document. There are m*k components to sum over in this case.

Russ