Help using Base SAS procedures

Clustering a large dataset with mixed variable types

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 5
Accepted Solution

Clustering a large dataset with mixed variable types

[ Edited ]

I have a dataset that has 700,000 rows and various variables with mixed data-types: categorical, numeric and binary. I have read several suggestions on how to cluster categorical data but still couldn't find a solution for my problem. The common suggestions are listed below:

1) Use proc distance for the categorical variables to get a distance matrix and then use proc cluster: I can't do that as my dataset is too big to be handled by proc cluster. Another constraint is that even proc fastclus can handle a large dataset but it doesn't work with distance matrix or anything other than numeric data.

2) I don't find "creating dummy variables for the categorical variables" a good solution either as I feel the clusters would be hard to interpret.

3) Another suggestion was to use HPCLUS but HPCLUS can either use only categorical variables or only numeric interval variables to perform clustering but it does not perform clustering for mixed levels of input variables

 

Is there any other solution/proc available in SAS that could perform clustering on a large dataset with mixed variable types? 


Accepted Solutions
Solution
‎08-21-2017 11:26 PM
Esteemed Advisor
Posts: 5,529

Re: Clustering a large dataset with mixed variable types

EM has HPCLUS which should handle larger problems.

PG

View solution in original post


All Replies
Super User
Posts: 23,724

Re: Clustering a large dataset with mixed variable types

How many variables do you have? 700k is a lot but I would have assumed SAS could handle it. Can you change your memory settings to allow it to be processed? 

Occasional Contributor
Posts: 5

Re: Clustering a large dataset with mixed variable types

There are 100+ variables in total but I will do variable selection before using cluster analysis. I might start with only 8-10 variables but the issue is I don't know how to do cluster analysis with binary/categorical variables.
Solution
‎08-21-2017 11:26 PM
Esteemed Advisor
Posts: 5,529

Re: Clustering a large dataset with mixed variable types

EM has HPCLUS which should handle larger problems.

PG
Occasional Contributor
Posts: 5

Re: Clustering a large dataset with mixed variable types

Can it handle mixed variable types too?
Esteemed Advisor
Posts: 5,529

Re: Clustering a large dataset with mixed variable types

Occasional Contributor
Posts: 5

Re: Clustering a large dataset with mixed variable types

Thank you so much!
☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 1745 views
  • 2 likes
  • 3 in conversation