Help using Base SAS procedures

choose between Proc cluster methods

Accepted Solution Solved
Reply
Frequent Contributor
Posts: 89
Accepted Solution

choose between Proc cluster methods

Hello!

I have a pretty large data set and have to cluster it. I have no logical assumption to prefer a specific number of clusters neither have a preference over the clustering method.

Considering the fact that each of the many available SAS clustering methods provides different clusters which may affect my subsequent analysis significantly, is there any method to evaluate the clustering methods to select the most appropriate one?  There is highly cited paper, which discusses this, but I can not find any SAS procedures or macros to perform the algorithms introduced in the paper

https://bitbucket.org/eyecat/readinglists/src/d8e8010f0b0d2dbb0863af3050411695254cc6b1/ReadingList_N...


Accepted Solutions
Solution
‎08-05-2013 10:09 AM
Respected Advisor
Posts: 2,655

Re: choose between Proc cluster methods

The "Introduction to Clustering Procedures" and many of the clustering proc's Reference sections cite Milligan (although not this particular paper) quite regularly.  Comparison of algorithms has been done extensively on simulated data, but not so much on real data as you don't have prior knowledge of the number of clusters.

Of the algorithms in the Milligan and Cooper paper, the FASTCLUS procedure implements the Calinski and Harabasz method, while the CLUSTER procedure implements an improved formula for the Duda and Hart method.

Good luck.

Steve Denham

View solution in original post


All Replies
Solution
‎08-05-2013 10:09 AM
Respected Advisor
Posts: 2,655

Re: choose between Proc cluster methods

The "Introduction to Clustering Procedures" and many of the clustering proc's Reference sections cite Milligan (although not this particular paper) quite regularly.  Comparison of algorithms has been done extensively on simulated data, but not so much on real data as you don't have prior knowledge of the number of clusters.

Of the algorithms in the Milligan and Cooper paper, the FASTCLUS procedure implements the Calinski and Harabasz method, while the CLUSTER procedure implements an improved formula for the Duda and Hart method.

Good luck.

Steve Denham

Frequent Contributor
Posts: 89

Re: choose between Proc cluster methods

Posted in reply to SteveDenham

Thank you very much for the info!

Contributor
Posts: 30

Re: choose between Proc cluster methods

Posted in reply to SteveDenham

Steve,

You mention that FASTCLUS procedure implements the Calinski and Harabasz method, but how to get it's value (is that readily available in the output). How to get value of c(g) =trace(B)/(g-1)/trace(W)/(n-g), where n and g are the total number of observation and number of clusters manually. Yyour help will be pivotal for me.

Thanks!

Contributor
Posts: 57

Re: choose between Proc cluster methods

The goodness of the coefficients depend on your data, the distance measure you use (euclidiean, city-block, correlation etc. ) and the kind of algorithmus that you choose. If you have no idea, how the data is going to look like, most of the people start with hierarchical clustering, especially average-linkage methode. the problem is the speed. it takes relativly longer time. if you have a reasonable guess, then k-means or K-medoids will be better, because they are pretty fast for large data sets. if you want more detailed information about the data sets, then try soft-clustering methode, like Fanny algo. In fast there is no the best methode. All the steps, like how you choose your distance measure, are not trivial.

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 987 views
  • 2 likes
  • 4 in conversation