topic Re: Using cross-validation in Enterprise Miner; in SAS Data Science

Using cross-validation in Enterprise Miner;

frak — Sat, 07 Nov 2015 16:02:23 GMT

Hi, this is my problem:

I've got a dataset on which i've to apply the technics of data mining.

Because my dataset is small (about 700 obs.) i thought not to use external validation, but cross-validation (10 fold);

Although i don't now how to make it on Enterprise Miner:

I mean, when i specify some models i can't find the way to tell Enterprise Miner to use cross-validation.

I E in trees models there is the explicit option "execute crossvalidation yes/no" but in other type models i can't find it.

So someone can help me to use cross-validation with regression and MBR models?

I hope i've written something you can understand.

Thank you

Re: Using cross-validation in Enterprise Miner;

M_Maldonado — Sat, 07 Nov 2015 16:56:50 GMT

Hi Frak,

You are right, three nodes that support crossvalidation directly are:

Decision tree
LARS regression
HPTree

For the rest of the model nodes (including some High-Performance model nodes), you can use the Start and End group nodes. As long as the node that you are using produces SAS score code, you can use it. This means most of the nodes (HP Regression, HP Tree, HPNeural, etc) will work well with this approach, but not HPForest or some settings of HPSVM.

Check out:

The Power of the Group Processing Facility in SAS® Enterprise Miner™, Sascha Schubert, SAS Institute Inc., Cary, NC

https://support.sas.com/resources/papers/proceedings10/123-2010.pdf

This paper walks you through all the cool things you can do with the Start and End group nodes. Bottomline, you need to use a Transform node to create a crossvalidation variable. Include that, and you are pretty much setup!

Good luck!

Miguel

[Edit: I added HPTree to the list. You can do crossvalidation cost-complexity when you do not have a Partition node]

Re: Using cross-validation in Enterprise Miner;

frak — Sat, 07 Nov 2015 16:27:15 GMT

Thank you Miguel !

Re: Using cross-validation in Enterprise Miner;

frak — Mon, 09 Nov 2015 17:37:52 GMT

Daaa Daan I'm back again, and i've new dubts..

As Miguel, (thank you again Miguel!) recommend me, I used Start and End node to obtain cross-validation.

So, as expected ,using a 10 fold crossvalidation, i obtained 11 different dataset (10 with 9/10 of data and 1 complete), for each of which EM calculated a model. That's OK.

BUT, in my studies I learnt (maybe) that also in k-fold cross-validation i've finally a validation dataset, which is the result of the "sum" of scores of each model (created on (k-1)/k of data) on the ramaining 1/k of data, AND this doesn't happen in EM: so i don't have a validation dataset.

This is a big problem for me, because, as I understood, i need a validation to evaluate the presence of overfiffing in my model, don't I? How could i tell if there's overfitting or not only looking training datasets?

Someone could tell me what is wrong?

Again, sorry for my terrible english, and thank you in advance!

Re: Using cross-validation in Enterprise Miner;

M_Maldonado — Mon, 09 Nov 2015 17:49:16 GMT

Hey bud,

Glad you are back. I'm the same, I always use a test set.

At the beginning of the your flow do: Data->Data Partition (70%training, 0% validation, 30%test)->then your crossvalidation flow->then a Model Comparison node [even if you only have one model, you are using it to see more fit statistics about your model].

Keep up the good work!

-Miguel

Re: Using cross-validation in Enterprise Miner;

frak — Mon, 09 Nov 2015 18:19:12 GMT

Oh, it seems you were waiting for me ahah .. so fast 🙂

Yes, i could use a data partitioning node but i would lose the benefits of crossvalidation: i would like not to divide my dataset...

..in any case thanks you a lot 🙂