BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
frak
Calcite | Level 5

Hi, this is my problem:

 

I've got a dataset on which i've to apply the technics of data mining.

Because my dataset is small (about 700 obs.) i thought not to use external validation, but cross-validation (10 fold);

Although i don't now how to make it on Enterprise Miner:

I mean, when i specify some models i can't find the way to tell Enterprise Miner to use cross-validation.

I E in trees models there is the explicit option "execute crossvalidation yes/no" but in other type models i can't find it.

So someone can help me to use cross-validation with regression and MBR models?

 

I hope i've written something you can understand.

Thank you 

1 ACCEPTED SOLUTION

Accepted Solutions
M_Maldonado
Barite | Level 11

Hi Frak,

You are right, three nodes that support crossvalidation directly are:

  • Decision tree
  • LARS regression
  • HPTree

For the rest of the model nodes (including some High-Performance model nodes), you can use the Start and End group nodes. As long as the node that you are using produces SAS score code, you can use it. This means most of the nodes (HP Regression, HP Tree, HPNeural, etc) will work well with this approach, but not HPForest or some settings of HPSVM.

 

Check out:

 

The Power of the Group Processing Facility in SAS® Enterprise Miner™, Sascha Schubert, SAS Institute Inc., Cary, NC

https://support.sas.com/resources/papers/proceedings10/123-2010.pdf

 

This paper walks you through all the cool things you can do with the Start and End group nodes. Bottomline, you need to use a Transform node to create a crossvalidation variable. Include that, and you are pretty much setup!

 

Good luck!

Miguel

 

[Edit: I added HPTree to the list. You can do crossvalidation cost-complexity when you do not have a Partition node]

View solution in original post

5 REPLIES 5
M_Maldonado
Barite | Level 11

Hi Frak,

You are right, three nodes that support crossvalidation directly are:

  • Decision tree
  • LARS regression
  • HPTree

For the rest of the model nodes (including some High-Performance model nodes), you can use the Start and End group nodes. As long as the node that you are using produces SAS score code, you can use it. This means most of the nodes (HP Regression, HP Tree, HPNeural, etc) will work well with this approach, but not HPForest or some settings of HPSVM.

 

Check out:

 

The Power of the Group Processing Facility in SAS® Enterprise Miner™, Sascha Schubert, SAS Institute Inc., Cary, NC

https://support.sas.com/resources/papers/proceedings10/123-2010.pdf

 

This paper walks you through all the cool things you can do with the Start and End group nodes. Bottomline, you need to use a Transform node to create a crossvalidation variable. Include that, and you are pretty much setup!

 

Good luck!

Miguel

 

[Edit: I added HPTree to the list. You can do crossvalidation cost-complexity when you do not have a Partition node]

frak
Calcite | Level 5

Thank you Miguel !

frak
Calcite | Level 5

Daaa Daan I'm back again, and i've new dubts.. 

 

As Miguel, (thank you again Miguel!) recommend me, I used Start and End node to obtain cross-validation.

So, as expected ,using a 10 fold crossvalidation, i obtained 11 different dataset (10 with 9/10 of data and 1 complete), for each of which EM calculated a model. That's OK. 

 

BUT, in my studies I learnt (maybe) that also in k-fold cross-validation i've finally a validation dataset, which is the result of the "sum" of scores of each model (created on (k-1)/k of data) on the ramaining 1/k of data, AND this doesn't happen in EM: so i don't have a validation dataset. 

 

This is a big problem for me, because, as I understood, i need a validation to evaluate the presence of overfiffing in my model, don't I? How could i tell if there's overfitting or not only looking training datasets?

 

Someone could tell me what is wrong?

 

Again, sorry for my terrible english, and thank you in advance!

 

 

 

 

 

 

M_Maldonado
Barite | Level 11

Hey bud,

Glad you are back. I'm the same, I always use a test set.

 

At the beginning of the your flow do: Data->Data Partition (70%training, 0% validation, 30%test)->then your crossvalidation flow->then a Model Comparison node [even if you only have one model, you are using it to see more fit statistics about your model].

 

Keep up the good work!

-Miguel

frak
Calcite | Level 5

Oh, it seems you were waiting for me ahah .. so fast 🙂

Yes, i could use a data partitioning node but i would lose the benefits of crossvalidation: i would like not to divide my dataset... 

..in any case thanks you a lot 🙂

 

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 8957 views
  • 1 like
  • 2 in conversation