topic Re: What should be the Optimum Number of Cluster in SAS Data Science

What should be the Optimum Number of Cluster

arpitsharma27 — Wed, 17 Jun 2020 17:26:16 GMT

Team,

I have this:

%macro clustering_method(method=);
proc cluster data=cars method=&method. ccc outtree=tree_&method. noprint;
where type="Sports";
by type;
	var horsepower mpg_highway weight wheelbase;
run;
proc sort data=tree_&method. out=&method.(keep= type _ncl_ _ccc_ );
by type _ncl_ _ccc_ ;
where not missing(_ccc_);
run;

%mend;
%clustering_method(method=Average);
%clustering_method(method=median);
%clustering_method(method=centroid);
%clustering_method(method=mcquitty);
%clustering_method(method=ward);

data Have;
set Average 
	Median 
	Centroid 
	McQuitty 
	Ward 
		indsname=source;
input_ds=scan(source,2,'.');;
run;

Referring to the Have dataset. What should be my optimum number of Clusters ? and Why?

Please advise.

Thanks

Re: What should be the Optimum Number of Cluster

PaigeMiller — Wed, 17 Jun 2020 18:28:14 GMT

You are getting only negative numbers for CCC. This implies (to me) that there is no clustering. Also see https://www.researchgate.net/post/Could_someone_help_me_decide_the_ideal_noof_clusters_from_the_pseudo_t_squared_graph_in_SAS

which says

CCC is the cubic clustering criterion; the idea behind it is to compare the R squared you get with a specific number of clusters versus the R squared you would get by clustering a uniformly distributed set of points. That is, you interpret it similarly as you would R squared. You are getting STRICTLY negative values (and, in fact, they are decreasing with additional number of clusters before increasing again; I would interpret that increase as overfitting). This means that the model you are fitting to the data with X number of clusters fits worse than uniformly distributed points. This is evidence of a lack of clustering (or problems with the data).

Re: What should be the Optimum Number of Cluster

Ksharp — Thu, 18 Jun 2020 12:19:15 GMT

It is a world unsolved problem.

If I were you, I would try Primary Component Analysis.

Anyway, @Rick_SAS maybe have some ideas .

Re: What should be the Optimum Number of Cluster

arpitsharma27 — Thu, 18 Jun 2020 13:24:19 GMT

Thank You @PaigeMiller & @Ksharp

I know _ccc_ is strictly negative.

I am already using PCA too.

The idea is to get outliers from 2 different algorithms and then join to get the output.

PCA-- was able to handle this.

But

KNN-- is looking for better selection of variables. Just dumping variables for KNN to figure out the cluster does not seem to be the correct thing to do.

Thank You to the Legends.

Re: What should be the Optimum Number of Cluster

MelodieRush — Thu, 18 Jun 2020 13:37:27 GMT

Aligned Box Criterion is available in the HP Cluster node in SAS Enterprise Miner. It will determine the optimum number of clusters. Here's a video that talks about using this option, along with using CCC and gap methods https://www.youtube.com/watch?v=NZpNTkfT47c