topic Re: Hierarchical clustering using distance = "spearman's correlation"? in SAS Programming

Hierarchical clustering using distance = "spearman's correlation"?

Rcore — Thu, 07 Jul 2016 20:13:15 GMT

In PROC CLUSTER or any other procedure, can I choose a distance method as a Spearman's correlation coefficient between variables?

If so, what is the option name?

Thank you very much for your help!

Re: Hierarchical clustering using distance = "spearman's correlation"?

PGStats — Thu, 07 Jul 2016 20:52:07 GMT

Correlations are proximity measures, not distance. You have to create your own distances from Spearman correlations as SQRT(1-S) and use them for clustering:

proc corr data=sashelp.class outs=os noprint;
var height weight age;
run;

data osd(type="DISTANCE");
set os;
where _TYPE_ = "CORR";
array d _NUMERIC_;
do i = 1 to dim(d);
    d{i} = sqrt(1 - d{i});
    end;
rename _NAME_ = ID;
drop _TYPE_ i;
run;

proc cluster data=osd method=flexible out=osdc;
id ID;
run;

Re: Hierarchical clustering using distance = "spearman's correlation"?

Rcore — Fri, 08 Jul 2016 17:00:05 GMT

Thank you very much for your kind response. Why did you choose "method=flexible" in proc cluster? Is this because the distance is based on correlation coefficients?

I've checked SAS User's Guid, but couldn't figure out.

I really appreciate your help!

Re: Hierarchical clustering using distance = "spearman's correlation"?

PGStats — Fri, 08 Jul 2016 18:48:16 GMT

Clustering is most often done among observations, not variables. Proc VARCLUS is the only SAS procedure for clustering variables but does not allow for user-defined distance measures. The method proposed is for clustering variables based on Spearman correlations with proc CLUSTER, a procedure which requires distances.

I have suggested PROC CLUSTER METHOD=FLEXIBLE because it has given me useful results for similar problems. That doesn't prevent you from exploring other options at your will. No clustering method is universally the best. I'd be interested to know what method you chose in the end.