BookmarkSubscribeRSS Feed
Babloo
Rhodochrosite | Level 12
I have done a few models (using random forest, decision tree and SVM) for one classification problem. Now I would like to compare the accuracy of those models. Would cross validation will help me to get the accuracy of those models or I have to go other validation technique?
5 REPLIES 5
PaigeMiller
Diamond | Level 26

Cross validation usually helps determine the precision (variability) of the estimates and the model.

 

If you are concerned about accuracy, you probably need to apply the model to a different sample (often called a hold-out sample or validation sample) and then determine how well the predicted classifications match the actual classification.

--
Paige Miller
Babloo
Rhodochrosite | Level 12
Ok, I'm curious to know when we shouldn't use the hold-out or validation
sample?
PaigeMiller
Diamond | Level 26

You don't use the validation sample when fitting the model. You use it when evaluating the model.

--
Paige Miller
Babloo
Rhodochrosite | Level 12
Now I came with a two final candidate models. One with logistic regression
which has a K-S of 39 percent and the other with SVM which has K-S of 41
percent. Which model you recommend me to deploy?
PaigeMiller
Diamond | Level 26

You (not me) have to look at these models and determine which one you like better, and which one works better for your situation. The best fitting model is not always chosen, as there could easily be other reasons why a model with a slightly worse fit makes more sense to use.

--
Paige Miller

sas-innovate-white.png

Special offer for SAS Communities members

Save $250 on SAS Innovate and get a free advance copy of the new SAS For Dummies book! Use the code "SASforDummies" to register. Don't miss out, May 6-9, in Orlando, Florida.

 

View the full agenda.

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 1266 views
  • 1 like
  • 2 in conversation