topic Re: Cross validation in Statistical Procedures

Cross validation

Babloo — Wed, 10 Apr 2019 09:31:33 GMT

I have done a few models (using random forest, decision tree and SVM) for one classification problem. Now I would like to compare the accuracy of those models. Would cross validation will help me to get the accuracy of those models or I have to go other validation technique?

Re: Cross validation

PaigeMiller — Wed, 10 Apr 2019 10:04:34 GMT

Cross validation usually helps determine the precision (variability) of the estimates and the model.

If you are concerned about accuracy, you probably need to apply the model to a different sample (often called a hold-out sample or validation sample) and then determine how well the predicted classifications match the actual classification.

Re: Cross validation

Babloo — Wed, 10 Apr 2019 10:09:18 GMT

Ok, I'm curious to know when we shouldn't use the hold-out or validation
sample?

Re: Cross validation

PaigeMiller — Wed, 10 Apr 2019 10:13:21 GMT

You don't use the validation sample when fitting the model. You use it when evaluating the model.

Re: Cross validation

Babloo — Thu, 11 Apr 2019 05:40:18 GMT

Now I came with a two final candidate models. One with logistic regression
which has a K-S of 39 percent and the other with SVM which has K-S of 41
percent. Which model you recommend me to deploy?

Re: Cross validation

PaigeMiller — Thu, 11 Apr 2019 12:03:18 GMT

You (not me) have to look at these models and determine which one you like better, and which one works better for your situation. The best fitting model is not always chosen, as there could easily be other reasons why a model with a slightly worse fit makes more sense to use.