New predictors decrease model accuracy

[ilya_1991]

Good day to everyone.

I'am using SAS RPM for modeling and got next problem:

I've builded model on some pull of predictors with Gini, say 0.5

Then I took same data, same predictors and added  new set of parameters, but unfortunately model became worse(lower gini).

In my understanding, if new predictor is "weak" and it can't improve model performance, it at least won't spoil it.

Does someone have any ideas?

[JBerry]

Your basic instinct "always add more data!"  is a good one. And you're sort of correct, that adding more data can't possibly make your situation worse because after all, you still have the original data, plus some more!  However - remember that RPM is just a machine. You are still trusting it to identify problems with the new data, and then back off. Remember that as you add more data, you might create a situation where your model is overfitting - meaning the model fits the training data better, but fits the holdout data worse. 

Now, if you were building the model by hand, you might notice that and say, "oh let me go back to my original model" But RPM will never be as smart as a human. To answer your question would require intimiate knowledge of your data and also how RPM works, of which I have neither.

Its hard to say exactly why your situation yielded a lower Gini. There are an infinite number of things that could have happened, but here is my best guess:

Either:

(a)  The model selection isn't using Gini but some other criteria  (perhaps the 2nd model produced an almost-as-good-gini with less variables and had a higher AIC)

(b) You added new variables which were all colinear with each other, but RPM only took the top N best. The new variables bumped off the original variables outside of the top N, so they weren't even considered in the 2nd model.

I'm sorry that I'm not an expert in RPM and can't answer your question directly, but I think you're on the right path. Youre adding more data, running models, and paying attention to details.

Good luck

[ilya_1991]

Thank you very much for your answer