choose between Proc cluster methods

[niam]

Hello!

I have a pretty large data set and have to cluster it. I have no logical assumption to prefer a specific number of clusters neither have a preference over the clustering method.

Considering the fact that each of the many available SAS clustering methods provides different clusters which may affect my subsequent analysis significantly, is there any method to evaluate the clustering methods to select the most appropriate one?  There is highly cited paper, which discusses this, but I can not find any SAS procedures or macros to perform the algorithms introduced in the paper

https://bitbucket.org/eyecat/readinglists/src/d8e8010f0b0d2dbb0863af3050411695254cc6b1/ReadingList_N...

[SteveDenham · Accepted solution]

The "Introduction to Clustering Procedures" and many of the clustering proc's Reference sections cite Milligan (although not this particular paper) quite regularly.  Comparison of algorithms has been done extensively on simulated data, but not so much on real data as you don't have prior knowledge of the number of clusters.

Of the algorithms in the Milligan and Cooper paper, the FASTCLUS procedure implements the Calinski and Harabasz method, while the CLUSTER procedure implements an improved formula for the Duda and Hart method.

Good luck.

Steve Denham

[niam]

Thank you very much for the info!

[koomalkc]

Steve,

You mention that FASTCLUS procedure implements the Calinski and Harabasz method, but how to get it's value (is that readily available in the output). How to get value of c(g) =trace(B)/(g-1)/trace(W)/(n-g), where n and g are the total number of observation and number of clusters manually. Yyour help will be pivotal for me.

Thanks!

[Dingdang]

The goodness of the coefficients depend on your data, the distance measure you use (euclidiean, city-block, correlation etc. ) and the kind of algorithmus that you choose. If you have no idea, how the data is going to look like, most of the people start with hierarchical clustering, especially average-linkage methode. the problem is the speed. it takes relativly longer time. if you have a reasonable guess, then k-means or K-medoids will be better, because they are pretty fast for large data sets. if you want more detailed information about the data sets, then try soft-clustering methode, like Fanny algo. In fast there is no the best methode. All the steps, like how you choose your distance measure, are not trivial.