Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Using cross-validation in Enterprise Miner;

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 7
Accepted Solution

Using cross-validation in Enterprise Miner;

Hi, this is my problem:

 

I've got a dataset on which i've to apply the technics of data mining.

Because my dataset is small (about 700 obs.) i thought not to use external validation, but cross-validation (10 fold);

Although i don't now how to make it on Enterprise Miner:

I mean, when i specify some models i can't find the way to tell Enterprise Miner to use cross-validation.

I E in trees models there is the explicit option "execute crossvalidation yes/no" but in other type models i can't find it.

So someone can help me to use cross-validation with regression and MBR models?

 

I hope i've written something you can understand.

Thank you 


Accepted Solutions
Solution
‎07-06-2017 02:26 PM
Super Contributor
Posts: 336

Re: Using cross-validation in Enterprise Miner;

[ Edited ]

Hi Frak,

You are right, three nodes that support crossvalidation directly are:

  • Decision tree
  • LARS regression
  • HPTree

For the rest of the model nodes (including some High-Performance model nodes), you can use the Start and End group nodes. As long as the node that you are using produces SAS score code, you can use it. This means most of the nodes (HP Regression, HP Tree, HPNeural, etc) will work well with this approach, but not HPForest or some settings of HPSVM.

 

Check out:

 

The Power of the Group Processing Facility in SAS® Enterprise Miner™, Sascha Schubert, SAS Institute Inc., Cary, NC

https://support.sas.com/resources/papers/proceedings10/123-2010.pdf

 

This paper walks you through all the cool things you can do with the Start and End group nodes. Bottomline, you need to use a Transform node to create a crossvalidation variable. Include that, and you are pretty much setup!

 

Good luck!

Miguel

 

[Edit: I added HPTree to the list. You can do crossvalidation cost-complexity when you do not have a Partition node]

View solution in original post


All Replies
Solution
‎07-06-2017 02:26 PM
Super Contributor
Posts: 336

Re: Using cross-validation in Enterprise Miner;

[ Edited ]

Hi Frak,

You are right, three nodes that support crossvalidation directly are:

  • Decision tree
  • LARS regression
  • HPTree

For the rest of the model nodes (including some High-Performance model nodes), you can use the Start and End group nodes. As long as the node that you are using produces SAS score code, you can use it. This means most of the nodes (HP Regression, HP Tree, HPNeural, etc) will work well with this approach, but not HPForest or some settings of HPSVM.

 

Check out:

 

The Power of the Group Processing Facility in SAS® Enterprise Miner™, Sascha Schubert, SAS Institute Inc., Cary, NC

https://support.sas.com/resources/papers/proceedings10/123-2010.pdf

 

This paper walks you through all the cool things you can do with the Start and End group nodes. Bottomline, you need to use a Transform node to create a crossvalidation variable. Include that, and you are pretty much setup!

 

Good luck!

Miguel

 

[Edit: I added HPTree to the list. You can do crossvalidation cost-complexity when you do not have a Partition node]

Occasional Contributor
Posts: 7

Re: Using cross-validation in Enterprise Miner;

Thank you Miguel !

Occasional Contributor
Posts: 7

Re: Using cross-validation in Enterprise Miner;

[ Edited ]

Daaa Daan I'm back again, and i've new dubts.. 

 

As Miguel, (thank you again Miguel!) recommend me, I used Start and End node to obtain cross-validation.

So, as expected ,using a 10 fold crossvalidation, i obtained 11 different dataset (10 with 9/10 of data and 1 complete), for each of which EM calculated a model. That's OK. 

 

BUT, in my studies I learnt (maybe) that also in k-fold cross-validation i've finally a validation dataset, which is the result of the "sum" of scores of each model (created on (k-1)/k of data) on the ramaining 1/k of data, AND this doesn't happen in EM: so i don't have a validation dataset. 

 

This is a big problem for me, because, as I understood, i need a validation to evaluate the presence of overfiffing in my model, don't I? How could i tell if there's overfitting or not only looking training datasets?

 

Someone could tell me what is wrong?

 

Again, sorry for my terrible english, and thank you in advance!

 

 

 

 

 

 

Super Contributor
Posts: 336

Re: Using cross-validation in Enterprise Miner;

[ Edited ]

Hey bud,

Glad you are back. I'm the same, I always use a test set.

 

At the beginning of the your flow do: Data->Data Partition (70%training, 0% validation, 30%test)->then your crossvalidation flow->then a Model Comparison node [even if you only have one model, you are using it to see more fit statistics about your model].

 

Keep up the good work!

-Miguel

Occasional Contributor
Posts: 7

Re: Using cross-validation in Enterprise Miner;

Oh, it seems you were waiting for me ahah .. so fast Smiley Happy

Yes, i could use a data partitioning node but i would lose the benefits of crossvalidation: i would like not to divide my dataset... 

..in any case thanks you a lot Smiley Happy

 

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 2113 views
  • 0 likes
  • 2 in conversation