Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

What is the best way to continuously train a SAS EM Model?

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 6
Accepted Solution

What is the best way to continuously train a SAS EM Model?

I am fairly new to SAS, but Im trying to build a model at work that will identify customers most likely to adopt a new product. We're making the process easily repeatable because we will be doing it monthly for a significant amount of time. My question is if we do this for several years, will the model go "out of tune" without being periodically re-trained with more recent data? If that's the case, should we make it part of our monthly process to train it with updated data? Or do we update it with a data set that has the three most recent month's worth of data? (Running SAS EM 12.3 and EG 6.1)

 

Any advice would be helpful.

 

Thanks.


Accepted Solutions
Solution
‎07-26-2016 02:03 PM
SAS Employee
Posts: 122

Re: What is the best way to continuously train a SAS EM Model?

Hi, This gets into model management and strategic planning. The triggering event for whether and when a model should be updated is your model performance. In other words, mechanically re-train likely is not necessary unless you have model performance data that supports the action. As for updating models, there is rebuild and there is recalibrate. Recalibration also breaks down to different practices. Sometime you can rebuild the model data set using the same old logic as the previous one, with newer data. If you don't see major profile changes on the new model data set, you can re-run the model, without changing any EDA parts, to see the model appear very differently. If you see major changes, you are actually rebuilding it. Sometime you have prior knowledge that the target population or your scoring data source has changed significantly, you should rebuild the model regardless of model performance. For example, one of your three data suppliers has stopped supplying data, now a set of key variables in your scoring equation have 1/3 missing values. And the situation appears persisting. You need to rebuild the model entirely. When you start to build up your model assets, also pay a bit attention to environmental factors. Such as seasonality. It is not uncommon that one has a winter model, summer model and spring model. Vs a model that fits all seasons by relying one or two season variables to automatically adjust, with the latter having stronger tendency to be unstable. One big edge you have with using EM is you get great version control, documentation and history of your models whichever way you play. Hope this helps? Thanks. Jason Xin

View solution in original post


All Replies
Solution
‎07-26-2016 02:03 PM
SAS Employee
Posts: 122

Re: What is the best way to continuously train a SAS EM Model?

Hi, This gets into model management and strategic planning. The triggering event for whether and when a model should be updated is your model performance. In other words, mechanically re-train likely is not necessary unless you have model performance data that supports the action. As for updating models, there is rebuild and there is recalibrate. Recalibration also breaks down to different practices. Sometime you can rebuild the model data set using the same old logic as the previous one, with newer data. If you don't see major profile changes on the new model data set, you can re-run the model, without changing any EDA parts, to see the model appear very differently. If you see major changes, you are actually rebuilding it. Sometime you have prior knowledge that the target population or your scoring data source has changed significantly, you should rebuild the model regardless of model performance. For example, one of your three data suppliers has stopped supplying data, now a set of key variables in your scoring equation have 1/3 missing values. And the situation appears persisting. You need to rebuild the model entirely. When you start to build up your model assets, also pay a bit attention to environmental factors. Such as seasonality. It is not uncommon that one has a winter model, summer model and spring model. Vs a model that fits all seasons by relying one or two season variables to automatically adjust, with the latter having stronger tendency to be unstable. One big edge you have with using EM is you get great version control, documentation and history of your models whichever way you play. Hope this helps? Thanks. Jason Xin
Occasional Contributor
Posts: 6

Re: What is the best way to continuously train a SAS EM Model?

Thanks. I think this does answer my question pretty well. We shouldn't need to re-develop the model unless we know of a significant change happening within the data, or we notice poor model performance.

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 2 replies
  • 317 views
  • 3 likes
  • 2 in conversation