Dear @Funda_SAS ,   Thanks for your quick reply. I am not sure whether I have completely understood your explanations (please see comments for 1 and 2):   1. Could I say that the "Train:AUC" is the average AUC for the training data across all 10 folds? Since am actually wondering how my models perform on the validation set, I would want to get the average AUC that has been achieved on the 10 holdouts (rather than the average on the training folds). In case "Train:AUC" is indeed the average AUC of the 10 training folds, how would I have to modify the structure to get the average AUC of the 10 holdouts?   2. So, could I interpret a lower cross validation error as a smaller variance of the model's performance (e.g. accuracy or AUC), similar to the standard deviation or not?   Thanks your for quick clarifications 🙂 ... View more

Hey there, Thanks for this useful explanation. I re-built the same structure for my data and it worked. Still, I have two questions regarding cross validation: 1. I would want to know how well my model(s) actually performed in terms of ROC (or misclassification rate as above) on average on the holdout sets. However, if I follow the structure above, the Model Comparison node only outputs the ROC (or misclassification) on train and not on the k test holdouts (correct?). How can I assess that? 2. What is the signification of cross validation error? What can I conclude from the fact that it is either small or large?   Any help would be highly appreciated! Thanks in advance. Best regards, Dario ... View more