topic Re: K-fold cross validation - is it applied at all? in SAS Studio

K-fold cross validation - is it applied at all?

yiyhio — Fri, 30 Jul 2021 07:36:53 GMT

Hi!

I have trained a model in model studio using a gradient boosting method with a 5-fold cross validation. However, on the results page, on the table that gives a summary of the amount of data used for training/validation, I cannot see the connection between this and my chosen 5-fold.

Here, I chose validation method

And when I look at the results table for the same node, it says that it is divided into approximately 60% and 30% for the training and validation set.

I was wondering what this means? Does the 5-fold cross validation not apply for some reason, or does this mean something else?

Thank you in advance!

Re: K-fold cross validation - is it applied at all?

sbxkoenk — Sat, 31 Jul 2021 19:51:21 GMT

Hello @yiyhio ,

This is cross-validation for assessing / selecting the model(s), not for constructing the model(s).

This is what the documentation says:

===========

Validation method — Specifies how to partition the data for assessing the models. Note that if your data is partitioned, then that partition is used and Validation method, Validation data proportion, and Cross validation number of folds are all ignored. Here are the possible values:
- Partition — Specifies using a single partition of a training set. With partition, you specify proportions to use for randomly assigning observations to each role.
- K-fold cross validation — Specifies using the k-fold cross validation method. In k-fold cross validation, each model evaluation requires k training executions (on k-1 data folds) and k scoring executions (on one holdout fold). This increases the evaluation time by approximately a factor of k.
For small to medium data tables, cross validation provides, on average, a better representation of error across the whole data table. Partition is the default value.

===========

To use k-fold cross-validation for constructing the model, see here:

Cross Validation of a Forest Model
https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/casactml/casactml_mltools_example01.htm

The above example uses the crossValidateML action (in PROC CAS).
The crossValidateML Action doc:
https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/casactml/casactml_mltools_details02.htm

Kind regards,

Koen

Re: K-fold cross validation - is it applied at all?

sbxkoenk — Sat, 31 Jul 2021 19:56:39 GMT