Easy way to test models

[matt23]

Hi.

I am a student and I'm working on regression models. I have 24 models for every hour with 7 variables. When I do the selection based on adjrsq AIC SBC and RMSE, I get couple models for each hour that are very close. Is there an easy way to check them or run them through the data to see how they would do ? And by easy, I mean something I would be able to do as I am only a student. Or should I just trust the selection options?

[PaigeMiller · Accepted solution]

That's the problem with fitting models. THere are many ways to measure how good the model fits, they don't always agree, and if you're fitting models once an hour in some automated fashion, you can't really spend the time trying to understand which model is really really really really the best. So, I think you need to pick a criteria for how good the model fits, and go with it. You might want to consult with statisticians at your company/university to see if they have a preference for certain model fit statistics in your specific application.

[matt23]

Thank you. Can I ask one more question?
Let's say my Y is larger on weekdays than it is on weekends. Can I assume that this is the reason why the slopes for my X variables are more sensitive during weekends ? For example, one of my X variable has slope -100 throughout the weekday, and ranges from -100 to -250 throughout the weekend. Can it be due to the the fact that the Y value is larger on the weekday therefore X variables have less impact?

[PaigeMiller]

That's one possible cause of the changing slopes, but if the Y values are just higher on the weekend, that could result in different intercepts rather than different slopes. It is not the only potential cause of higher slopes. The difference in slopes could be due to outliers on the weekends, or higher variability on the weekends, or (depending on what modeling you are doing) more collinearity on weekends, and probably a few other things that I can't think of right now.

Also, Y in the range -100 to -250 on weekends is lower then weekdays, not higher.