I have a dataset that has 700,000 rows and various variables with mixed data-types: categorical, numeric and binary. I have read several suggestions on how to cluster categorical data but still couldn't find a solution for my problem. The common suggestions are listed below:
1) Use proc distance for the categorical variables to get a distance matrix and then use proc cluster: I can't do that as my dataset is too big to be handled by proc cluster. Another constraint is that even proc fastclus can handle a large dataset but it doesn't work with distance matrix or anything other than numeric data.
2) I don't find "creating dummy variables for the categorical variables" a good solution either as I feel the clusters would be hard to interpret.
3) Another suggestion was to use HPCLUS but HPCLUS can either use only categorical variables or only numeric interval variables to perform clustering but it does not perform clustering for mixed levels of input variables
Is there any other solution/proc available in SAS that could perform clustering on a large dataset with mixed variable types?
How many variables do you have? 700k is a lot but I would have assumed SAS could handle it. Can you change your memory settings to allow it to be processed?
EM has HPCLUS which should handle larger problems.
The doc says it does, check
http://support.sas.com/documentation/onlinedoc/miner/em14/emhpprcref.pdf
Pg 82: "HPCLUS procedure uses only numeric interval variables or only nominal variables to perform clustering. It does not perform clustering for mixed levels of input variables."
My bad. Out of the HP parallel processing procedures I only use HPSPLIT which allows mixed types. I wrongly assumed that HPCLUS would do the same.
Thanks @brintonj for setting the record strait.
Don't miss out on SAS Innovate - Register now for the FREE Livestream!
Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.