SAS Registered Model - Retrain & Versioning

[utrocketeng]

Greetings Support Community,

... here goes my first post ... wish me luck!

First, here are the particulars of our environment:

  * SAS 9.4M4

  * EMiner 14.2

  * DI Stuido 4.902

My colleauge and i are working to productionalize an analytics life cycle using EG, EMiner, and DI Studio.  We are using EG to ETL datasets for use in EMiner.  EMiner to create our model(s) and register them.  DI Studio to score new data using the Registered Models.  

At this point, we have demonstrated the entire cycle as described above.  The issue i see now is that when we need to "retrain" a model as a test, EM created a new registered model with the same name (different Model Key).  

is the above expected behavior?

IF SO:

this presents a challenge, as DI Studio will required a users to go update 'Model' nodes to use the newly trained models.  then reset the output table from that node.  then map downstream nodes as necessary.  it is somewhat unpleasent when all we are tyring to do is use newly trained models.  

IF NOT:

any thoughts why this would occur?  suggestions to prevent this from occuring?  also, is it possible to extract additional METADATA about the model in DI Stuido.  such as model version, date created, etc...

Thank you so much for any feedback,

Thomas

[TomKari]

Apologies, I have minimal experience with EM, so I can't address the main question in your post.

However, after reading it I would like to suggest that now that your ETL process is production-ready, you move it from EG to DI studio. In my experience DI Studio is much more of a production environment, with EG being ideal for development and exploration.

I think you'll find over the long run that DI Studio supports producton operations in a more reliable manner.

Best,

   Tom

[LinusH]

Like @TomKari I don't have much knowledge about EM.

Just want to add that DI Studio mainly holds metadata about table structures, jobs, authorization and similar. Application specific metadata is better browsed from other applications. For Enterprise Miner models, I think that SAS Model Manager is what you are looking for...?

[utrocketeng]

Thank you!  i am in the process of learning Model manager now.  

[Patrick]

@utrocketeng

Just continue asking that well formulated questions. You probably should have posted it under the Analytics section in this forum as like the others I'm more of a data guy and don't have sufficient EM knowledge to answer your question (may be post a cross link to this question into the Analytics forum so it gets the attention of the right people).

I'm currently working in a project which will implement and support such a model life cycle (part of a SAS Risk solution) also including a supporting workflow. 

I don't know enough about this area yet to answer your question but this is certainly something where SAS got all the right tools and a lot of procedural knowledge and experience. I feel you've got the "perfect" question for contacting your local SAS office directly.

[AnnaBrown]

Good discussion here! I've cross-posted to the Data Mining and Machine Learning Community, as I was the one who moved from there to here in the first place. 🙂 

[jifwei · Accepted solution]

SAS Model Manager can fufill what you are looking for. The following is the steps:

1. Build a model using SAS Enterprise Miner. You can either create a model package or register  the model into SAS Metadata Repository.
2. Create a project using SAS Model Manager, then import the model you created in step 1 into SAS Model Manager either through the model package you created in step 1 or import the model from SAS Metata Repository you created in step 1.
3. You set the imported model as a champion model in SAS Model Manager, then you can publish the SAS Model Manager  project into SAS Metadata Repository. A Miningresult object is created in SAS Metadata Repository
4. Then you build a DI job using the Miningresult object you created in step 3 through Model scoring transformation.
5. SAS Model Manager allows you to retrain the SAS EnterpriseMiner Model  you created in step 1 and register the newly retrained model back to the same SAS Model Manager project. If the newly retrained model performs better than the previous model, then you can set the newly retrained model as a champion model. Alternativley, you can also repeat  step 1 and step 2 to get the newly retrained model into the same SAS Model Manager project.
6. Then you publish the SAS Model Project again into the SAS Metadata Repository. SAS Model Manager will update the same Miningresult object you created in step 3. The score code for the Miningresult object in SAS Metadata Repository will be updated with the code from the latest champion model from the SAS Model Manager project.
7. Then you can refresh your DI job and rerun the job.

Hopefully this can get what you want. For your reference, you can check the SAS Model Manager documentation here: http://support.sas.com/documentation/onlinedoc/modelmgr/index.html

[utrocketeng]

Thank You Jifwei,

this is exactly the path i've been walking down using the very resources you linked.  i appreciate your order list of activites, it confirms that i'd been moving in the right direction.

Thanks Again,

thomas

[arindam1984]

Hi,

This is a good summary!

However how can I make Model Manager automatically retrain and replace the new model in metadata repository?

Thanks

Arindam