turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Unexpected clusters from PROC CORR data using PROC...

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

09-03-2016 09:00 AM

I would like to discover clusters of simple line plots. I ran CORR on the plots and subtracted the correlations from 1 to get "distances" between each plot.

I was surprised to see that CLUSTER did not always provide low level clusters of the closest plots with any of the methods that I tried. I expect that this is because CLUSTER treats each column sort of as a position in an 'n space' dimension. i.e. it does not rely on the distance calculated by CORR between 2 plots to determine the distance to use and doesn't know that column names match id variable values. I tried Type=DISTANCE as well with no success, though I can't claim to understand how distance is treated differently from coordinates.

The X axis range for the plots varies so the overlap between plots is inconsistent which may be what allows 2 highly correlated plots to have more variability in correlations with less related plots.

I was hoping to find small clusters of the most correlated plots that then comprise larger clusters, and so on. Is there a way to do that? Or do I need to code it myself using the agglomerate paradigm? Or am I doing something dumb?

I'm no expert at clustering so I wouldn't be surprised to find I have a conceptual issue.

Note CORR reports VADAX and MVCAX are the 2nd most correlated plot pair, but they do not comprise a low level cluster.

FWIW SAS 3.5 University Edition

Thanks, Duane

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

09-09-2016 03:47 AM

Your assuptions are correct: By default proc cluster "treats each column sort of as a position in an 'n space' dimension".

And yes, you have to use type=distance to change this behavior.

The trick is, that when you use **id LeftName;** only the rows in the distance matrix are identified.

The column names are ignored! In the distance matrix the columns must be in the same order as the rows!

Code solution:

**proc sort data=lib.Corr1;**** by leftName rightName;****run;**

proc transpose data=lib.Corr1

out=lib.Corr1T;

by leftName;

id rightName;

run;

/*Now all the 0-s are in the diagonal of the distance matrix*/

proc cluster data=lib.Corr1T(type=distance)

outtree=lib.ClusterTree

method=average nosquare;

id LeftName;

run;

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

09-09-2016 12:55 PM

Thanks a lot. That's very helpful. Duane