topic Re: Process New Data Set with Clustering Model in Statistical Procedures

Process New Data Set with Clustering Model

bkokster — Wed, 08 Jun 2016 04:46:54 GMT

I've created some clusters using proc fastclus, and I would like to apply the same clustering rulesets to a new dataset.

I'm assuming that this can be done by measuring the distance of each observation to the cluster centroids, and classifying each observation into it's nearest cluster.

Is there a built in function that allows for this?

Thanks in advance.

Re: Process New Data Set with Clustering Model

Ksharp — Wed, 08 Jun 2016 06:52:10 GMT

No. you can't . It lead you to another Statistical Analysis Approach -- DISCRIM Procedure .

Why not use Logistic Regression or Decision Tree ? both could handle character and numeric variables , while proc fastclus can't .

Re: Process New Data Set with Clustering Model

Rick_SAS — Wed, 08 Jun 2016 13:40:54 GMT

In PROC FASTCLUS you can use the OUTSTAT= option to create an output data set that contains the centers of each cluster:

proc fastclus data=sashelp.iris maxclusters=3 outstat=OutClus;
VAR SepalLength SepalWidth PetalLength PetalWidth;
run;

If you have some new data, you can merge the new data with the centers and use PROC DISTANCE to compute the distance between the new obs and the centers. the followin uses Euclidean distance, but other distances are available:


data NewData;
input SepalLength SepalWidth PetalLength PetalWidth;
datalines;
50 33 14 2 
62 22 45 15 
59 32 48 18 
64 28 56 22 
67 31 56 24 
;

data ALL;
set OutClus(where=(_TYPE_="CENTER")) NewData;
run;

proc distance data=All prefix=DistFromClus_ out=distances;
var interval(SepalLength SepalWidth PetalLength PetalWidth);
copy Cluster;
run;

/* we only want distances between centers */
data distances;
set distances;
where cluster = .;
array arr[*] DistFromClus_1-DistFromClus_3;
ClusterID = whichn(min(of arr[*]), of arr[*]); /* find index of min value in row */
keep DistFromClus_1-DistFromClus_3 ClusterID;
run;

proc print; run;

Re: Process New Data Set with Clustering Model

bkokster — Thu, 09 Jun 2016 05:05:47 GMT

Thanks Rick_SAS, that answers my question!

Re: Process New Data Set with Clustering Model

Ksharp — Thu, 09 Jun 2016 12:03:35 GMT

Rick,
If that should be , Why not combine them together and then run proc fastclus ?

data want;
 set sashelp.iris  new_date;
run;

proc fastclus data=want .....

Re: Process New Data Set with Clustering Model

Rick_SAS — Thu, 09 Jun 2016 12:22:00 GMT

I interpret the question as "I want to build a model on Data A and then score the model on Data B." If you merge the data, then you are using the second set of obs to build the model, which is not the same thing.

Re: Process New Data Set with Clustering Model

bkokster — Thu, 09 Jun 2016 12:27:37 GMT

Yes that's right. Using your example: Data A is consumer sales for the past month that will be used to cluster consumers. The client will compile a number of strategies around these clusters. At the end of the following month, Data B will be scored with the clusters developed on Data A. So we'll need to understand if there were any shifts in the clusters that were developed on Data A. Combining Data A and Data B and reclustering could result in totally different clusters and mess with the client's strategies....

Re: Process New Data Set with Clustering Model

Ksharp — Thu, 09 Jun 2016 12:41:49 GMT

Rick, Discrim Analysis do the exact same thing as "I want to build a model on Data A and then score the model on Data B." So I suggest OP to use PROC DISCRM .

Re: Process New Data Set with Clustering Model

Rick_SAS — Thu, 09 Jun 2016 12:49:57 GMT

Sorry, KSharp, but I disagree. Discriminant analysis is an example of supervised learning. You need a nominal target variable Y with k levels and the goal is to group the explanatory variables X into k groups so that most of Group1 has Y=1, most of Group2 has Y=2, etc.

Clustering is a form of unsupervised learning. The OP did not mention a target variable. In unsupervised learning you only have X and you want to group the observations together, often by using some distance metric.

Re: Process New Data Set with Clustering Model

bkokster — Thu, 09 Jun 2016 12:52:06 GMT

Correct, this is an exploratory analysis. The client simply wants to know who their clients are, there is no target to be modelled/predicted.....

Re: Process New Data Set with Clustering Model

Ksharp — Thu, 09 Jun 2016 13:03:45 GMT

Rick, Yeah. As you said Clustering Analysis only need TRAIN table, don't need TEST table. while Discriminant analysis need both TRAIN and TEST table. That is the most different thing between them. Since OP don't have Y variable( don't know which obs belong to which Y ). Why not combine them together and let PROC FASTCLUS tell you ?

Re: Process New Data Set with Clustering Model

Ksharp — Thu, 09 Jun 2016 13:16:22 GMT

If OP make sure which obs belong to which Y , Why not use Discriminant analysis?