BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
jordanruegg
Fluorite | Level 6

I am fairly new to SAS, but Im trying to build a model at work that will identify customers most likely to adopt a new product. We're making the process easily repeatable because we will be doing it monthly for a significant amount of time. My question is if we do this for several years, will the model go "out of tune" without being periodically re-trained with more recent data? If that's the case, should we make it part of our monthly process to train it with updated data? Or do we update it with a data set that has the three most recent month's worth of data? (Running SAS EM 12.3 and EG 6.1)

 

Any advice would be helpful.

 

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions
JasonXin
SAS Employee
Hi, This gets into model management and strategic planning. The triggering event for whether and when a model should be updated is your model performance. In other words, mechanically re-train likely is not necessary unless you have model performance data that supports the action. As for updating models, there is rebuild and there is recalibrate. Recalibration also breaks down to different practices. Sometime you can rebuild the model data set using the same old logic as the previous one, with newer data. If you don't see major profile changes on the new model data set, you can re-run the model, without changing any EDA parts, to see the model appear very differently. If you see major changes, you are actually rebuilding it. Sometime you have prior knowledge that the target population or your scoring data source has changed significantly, you should rebuild the model regardless of model performance. For example, one of your three data suppliers has stopped supplying data, now a set of key variables in your scoring equation have 1/3 missing values. And the situation appears persisting. You need to rebuild the model entirely. When you start to build up your model assets, also pay a bit attention to environmental factors. Such as seasonality. It is not uncommon that one has a winter model, summer model and spring model. Vs a model that fits all seasons by relying one or two season variables to automatically adjust, with the latter having stronger tendency to be unstable. One big edge you have with using EM is you get great version control, documentation and history of your models whichever way you play. Hope this helps? Thanks. Jason Xin

View solution in original post

2 REPLIES 2
JasonXin
SAS Employee
Hi, This gets into model management and strategic planning. The triggering event for whether and when a model should be updated is your model performance. In other words, mechanically re-train likely is not necessary unless you have model performance data that supports the action. As for updating models, there is rebuild and there is recalibrate. Recalibration also breaks down to different practices. Sometime you can rebuild the model data set using the same old logic as the previous one, with newer data. If you don't see major profile changes on the new model data set, you can re-run the model, without changing any EDA parts, to see the model appear very differently. If you see major changes, you are actually rebuilding it. Sometime you have prior knowledge that the target population or your scoring data source has changed significantly, you should rebuild the model regardless of model performance. For example, one of your three data suppliers has stopped supplying data, now a set of key variables in your scoring equation have 1/3 missing values. And the situation appears persisting. You need to rebuild the model entirely. When you start to build up your model assets, also pay a bit attention to environmental factors. Such as seasonality. It is not uncommon that one has a winter model, summer model and spring model. Vs a model that fits all seasons by relying one or two season variables to automatically adjust, with the latter having stronger tendency to be unstable. One big edge you have with using EM is you get great version control, documentation and history of your models whichever way you play. Hope this helps? Thanks. Jason Xin
jordanruegg
Fluorite | Level 6

Thanks. I think this does answer my question pretty well. We shouldn't need to re-develop the model unless we know of a significant change happening within the data, or we notice poor model performance.

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 2 replies
  • 1037 views
  • 3 likes
  • 2 in conversation