topic Re: What procedure is best for clustering? in Statistical Procedures

What procedure is best for clustering?

pkmkart — Fri, 24 Apr 2015 17:55:33 GMT

I have data which is very big (more than million rows) and it is not normally distributed. It has the following variables

Id (numeric)

form id (text)

product code (text)

sub-product code

test number (text)

test code (text)

collection date (numeric)

state (text)

district (text)

result (text)

result (numeric) (which has more negative results and very few positive result)

and other variables which are not used for analysis or it contains null values.

My tasks is to analysis this data with cluster procedures (fastclus, varclus and others ), but I am not very sure that which procedure will best suit my data, moreover I am dealing with the clustering for the first time.

My objective is to create clusters for positive count and based on the geographic location(state) and the product code or the collection date?From documentation I understood that fastclus may be good, is that correct ?

Any suggestions are welcome!!

Thanks In Advance

Re: What procedure is best for clustering?

Reeza — Fri, 24 Apr 2015 19:23:56 GMT

Do your basic univariate analysis first. It looks like you have mostly categorical variables. You show a 'result' variable, is that results from an experiment or a value that you're trying to cluster on that is independent.

I'd even consider regrouping the date into months or weeks. A million rows isn't really big data.

Be careful with using data that is related, i.e. product code and sub product code in the same analysis.

Re: What procedure is best for clustering?

plf515 — Tue, 28 Apr 2015 11:23:20 GMT

I don't understand what you are clustering. If "ID" is, indeed, an ID number, then it seems unlikely (to me) to be usefully included in any cluster analysis. Unless ID has some meaning beyond just being a code - but most ID numbers, even if they do have such meaning - don't relate linearly to anything.

But what are all the variables? What is the ID the identification for? A customer? Is each unique?

What are you trying to do? Not "I'm trying to cluster these data" but, in a practical, real-world sense, what is the purpose?

I wrote a blog post called How to Ask a Statistics Question that may be useful

Message was edited by: Peter Flom

Re: What procedure is best for clustering?

PaigeMiller — Tue, 28 Apr 2015 12:35:35 GMT

I have to agree with , this problem description seems to be very non-informative

My objective is to create clusters for positive count

What does "positive count" mean?

Clusters of what? Variables? Subjects? I don't see how this data, as you described it, can produce clusters anyway, the only real numeric variable is "result", the others ("id" and "collection date") are not meaningful variables, they are just identifiers.

People just don't create clusters for no reason. Explain what the reason is that you want these clusters, and how the list of variables can possibly produce clusters that are useful to the underlying reason for clusters.

Re: What procedure is best for clustering?

PaigeMiller — Tue, 28 Apr 2015 12:37:12 GMT

The link to your blog post is incorrect

Re: What procedure is best for clustering?

plf515 — Tue, 28 Apr 2015 12:46:03 GMT

Hi Paige

Not sure what happened. I am at SGF and a little busy but i will try to fix it ASAP

If you Google "How to ask a statistics question" and my name you can find it

Peter