04-24-2015 01:55 PM
I have data which is very big (more than million rows) and it is not normally distributed. It has the following variables
form id (text)
product code (text)
test number (text)
test code (text)
collection date (numeric)
result (numeric) (which has more negative results and very few positive result)
and other variables which are not used for analysis or it contains null values.
My tasks is to analysis this data with cluster procedures (fastclus, varclus and others ), but I am not very sure that which procedure will best suit my data, moreover I am dealing with the clustering for the first time.
My objective is to create clusters for positive count and based on the geographic location(state) and the product code or the collection date?From documentation I understood that fastclus may be good, is that correct ?
Any suggestions are welcome!!
Thanks In Advance
04-24-2015 03:23 PM
Do your basic univariate analysis first. It looks like you have mostly categorical variables. You show a 'result' variable, is that results from an experiment or a value that you're trying to cluster on that is independent.
I'd even consider regrouping the date into months or weeks. A million rows isn't really big data.
Be careful with using data that is related, i.e. product code and sub product code in the same analysis.
04-28-2015 07:23 AM
I don't understand what you are clustering. If "ID" is, indeed, an ID number, then it seems unlikely (to me) to be usefully included in any cluster analysis. Unless ID has some meaning beyond just being a code - but most ID numbers, even if they do have such meaning - don't relate linearly to anything.
But what are all the variables? What is the ID the identification for? A customer? Is each unique?
What are you trying to do? Not "I'm trying to cluster these data" but, in a practical, real-world sense, what is the purpose?
I wrote a blog post called How to Ask a Statistics Question that may be useful
Message was edited by: Peter Flom
04-28-2015 08:35 AM
My objective is to create clusters for positive count
What does "positive count" mean?
Clusters of what? Variables? Subjects? I don't see how this data, as you described it, can produce clusters anyway, the only real numeric variable is "result", the others ("id" and "collection date") are not meaningful variables, they are just identifiers.
People just don't create clusters for no reason. Explain what the reason is that you want these clusters, and how the list of variables can possibly produce clusters that are useful to the underlying reason for clusters.
04-28-2015 08:46 AM
Not sure what happened. I am at SGF and a little busy but i will try to fix it ASAP
If you Google "How to ask a statistics question" and my name you can find it