Programming the statistical procedures from SAS

Process New Data Set with Clustering Model

Accepted Solution Solved
Reply
New Contributor
Posts: 4
Accepted Solution

Process New Data Set with Clustering Model

I've created some clusters using proc fastclus, and I would like to apply the same clustering rulesets to a new dataset.

I'm assuming that this can be done by measuring the distance of each observation to the cluster centroids, and classifying each observation into it's nearest cluster. 

Is there a built in function that allows for this?

Thanks in advance.


Accepted Solutions
Solution
‎06-09-2016 01:06 AM
SAS Super FREQ
Posts: 3,547

Re: Process New Data Set with Clustering Model

In PROC FASTCLUS you can use the OUTSTAT= option to create an output data set that contains the centers of each cluster:

 

proc fastclus data=sashelp.iris maxclusters=3 outstat=OutClus;
VAR SepalLength SepalWidth PetalLength PetalWidth;
run;

If you have some new data, you can merge the new data with the centers and use PROC DISTANCE to compute the distance between the new obs and the centers.  the followin uses Euclidean distance, but other distances are available:

 

 


data NewData;
input SepalLength SepalWidth PetalLength PetalWidth;
datalines;
50 33 14 2 
62 22 45 15 
59 32 48 18 
64 28 56 22 
67 31 56 24 
;

data ALL;
set OutClus(where=(_TYPE_="CENTER")) NewData;
run;

proc distance data=All prefix=DistFromClus_ out=distances;
var interval(SepalLength SepalWidth PetalLength PetalWidth);
copy Cluster;
run;

/* we only want distances between centers */
data distances;
set distances;
where cluster = .;
array arr[*] DistFromClus_1-DistFromClus_3;
ClusterID = whichn(min(of arr[*]), of arr[*]); /* find index of min value in row */
keep DistFromClus_1-DistFromClus_3 ClusterID;
run;

proc print; run;

 

 

View solution in original post


All Replies
Super User
Posts: 9,775

Re: Process New Data Set with Clustering Model

No. you can't . It lead you to another Statistical Analysis Approach -- DISCRIM Procedure  .

Why not use Logistic Regression or Decision Tree ? both could handle character and numeric variables , while proc fastclus can't .

Solution
‎06-09-2016 01:06 AM
SAS Super FREQ
Posts: 3,547

Re: Process New Data Set with Clustering Model

In PROC FASTCLUS you can use the OUTSTAT= option to create an output data set that contains the centers of each cluster:

 

proc fastclus data=sashelp.iris maxclusters=3 outstat=OutClus;
VAR SepalLength SepalWidth PetalLength PetalWidth;
run;

If you have some new data, you can merge the new data with the centers and use PROC DISTANCE to compute the distance between the new obs and the centers.  the followin uses Euclidean distance, but other distances are available:

 

 


data NewData;
input SepalLength SepalWidth PetalLength PetalWidth;
datalines;
50 33 14 2 
62 22 45 15 
59 32 48 18 
64 28 56 22 
67 31 56 24 
;

data ALL;
set OutClus(where=(_TYPE_="CENTER")) NewData;
run;

proc distance data=All prefix=DistFromClus_ out=distances;
var interval(SepalLength SepalWidth PetalLength PetalWidth);
copy Cluster;
run;

/* we only want distances between centers */
data distances;
set distances;
where cluster = .;
array arr[*] DistFromClus_1-DistFromClus_3;
ClusterID = whichn(min(of arr[*]), of arr[*]); /* find index of min value in row */
keep DistFromClus_1-DistFromClus_3 ClusterID;
run;

proc print; run;

 

 

New Contributor
Posts: 4

Re: Process New Data Set with Clustering Model

Thanks Rick_SAS, that answers my question!

Super User
Posts: 9,775

Re: Process New Data Set with Clustering Model

Rick,
If that should be , Why not combine them together and then run proc fastclus ?

data want;
 set sashelp.iris  new_date;
run;

proc fastclus data=want .....


SAS Super FREQ
Posts: 3,547

Re: Process New Data Set with Clustering Model

I interpret the question as "I want to build a model on Data A and then score the model on Data B."  If you merge the data, then you are using the second set of obs to build the model, which is not the same thing.

 

New Contributor
Posts: 4

Re: Process New Data Set with Clustering Model

Yes that's right. Using your example: Data A is consumer sales for the past month that will be used to cluster consumers. The client will compile a number of strategies around these clusters. At the end of the following month, Data B will be scored with the clusters developed on Data A. So we'll need to understand if there were any shifts in the clusters that were developed on Data A. Combining Data A and Data B and reclustering could result in totally different clusters and mess with the client's strategies....

Super User
Posts: 9,775

Re: Process New Data Set with Clustering Model

Rick, Discrim Analysis do the exact same thing as "I want to build a model on Data A and then score the model on Data B." So I suggest OP to use PROC DISCRM .
SAS Super FREQ
Posts: 3,547

Re: Process New Data Set with Clustering Model

Sorry, KSharp, but I disagree. Discriminant analysis is an example of supervised learning. You need a nominal target variable Y with k levels and the goal is to group the explanatory variables X into k groups so that most of Group1 has Y=1, most of Group2 has Y=2, etc.

 

Clustering is a form of unsupervised learning. The OP did not mention a target variable. In unsupervised learning you only have X and you want to group the observations together, often by using some distance metric.

New Contributor
Posts: 4

Re: Process New Data Set with Clustering Model

Correct, this is an exploratory analysis. The client simply wants to know who their clients are, there is no target to be modelled/predicted.....

Super User
Posts: 9,775

Re: Process New Data Set with Clustering Model

Rick, Yeah. As you said Clustering Analysis only need TRAIN table, don't need TEST table. while Discriminant analysis need both TRAIN and TEST table. That is the most different thing between them. Since OP don't have Y variable( don't know which obs belong to which Y ). Why not combine them together and let PROC FASTCLUS tell you ?
Super User
Posts: 9,775

Re: Process New Data Set with Clustering Model

If OP make sure which obs belong to which Y , Why not use Discriminant analysis?
☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 11 replies
  • 380 views
  • 4 likes
  • 3 in conversation