topic Re: a rare target using an oversimple approach in SAS Enterprise Miner in SAS Data Science

a rare target using an oversimple approach in SAS Enterprise Miner

sathya66 — Mon, 26 Sep 2016 20:36:49 GMT

Hi ,

My data source target variable has (~0.0019%) and overall there are 7,240,251 observations,please can you explain through oversample approach(step by step please) or is there any other method to build a model.

I am using sample node now but it seems not working for me

Thanks,

Sathya.

Re: a rare target using an oversimple approach in SAS Enterprise Miner

WendyCzika — Tue, 27 Sep 2016 14:55:42 GMT

Please take a look at this post: https://communities.sas.com/t5/SAS-Communities-Library/Tip-How-to-model-a-rare-target-using-an-oversample-approach-in/ta-p/223599. This outlines one of the ways you can deal with rare targets. There is also a section "Detecting Rare Classes" under Analytics>Predictive Modeling in the SAS Enterprise Miner Reference Help.

Hope this helps!

Wendy

Re: a rare target using an oversimple approach in SAS Enterprise Miner

sathya66 — Tue, 27 Sep 2016 15:34:31 GMT

Thanks Wendy,
I have been through those steps but no luck.
EM_EVENTPROBABILITY coulmn values are >0.50 for 2,41,000 rows ( means>50%).
but SAS eminer will give us less %'s (ex :-0.00678) I guess
please suggest me
Thanks,
sathya.

Re: a rare target using an oversimple approach in SAS Enterprise Miner

sathya66 — Tue, 27 Sep 2016 15:55:47 GMT

and also after scoring ,it is creating some variables ex:
EM_EVENTPROBABILITY
EM_PROBABILITY
.EM_CLASSIFICATION (in 0s and 1s),
which variable do I need to consider from them as a predicted column
Thanks,
Sathya.

Re: a rare target using an oversimple approach in SAS Enterprise Miner

WendyCzika — Tue, 27 Sep 2016 18:20:15 GMT

So actually I think you want to be using the other approach for dealing with rare targets, which is to adjust the posterior probabilities instead of entering the decision weights (those only affect profit, not other fit statistics). So do that, in the Decisions node, you would no longer use the inverse priors on the diagonal of the decision matrix but just revert those to 1's, then you want to click Refresh on the Targets tab, then on the Prior Probabilities tab, enter the original priors for your target (the very rare proportion for your event, e.g.). Now this will apply an adjustment to your posterior probabilities - hopefully you will see better results this way.

To answer your other question, EM_CLASSIFICATION is the generically named variable containing the predictions based on your model. Here are more details about those variables from the Score node:

EM_PROBABILITY	Probability of Classification	Posterior probability associated with the predicted classification. That is, it corresponds the maximum of the posterior probabilities, max(P1, P2, ..., Pk).
EM_EVENTPROBABILITY	Probability for level n of vnm	Posterior probability associated with target event.
EM_CLASSIFICATION	Prediction for vnm	I_variable, the prediction variable for a class target.

Re: a rare target using an oversimple approach in SAS Enterprise Miner

sathya66 — Mon, 03 Oct 2016 11:21:34 GMT

Thank you Wendy.

Re: a rare target using an oversimple approach in SAS Enterprise Miner

sathya66 — Wed, 05 Oct 2016 08:18:40 GMT

And now I built a model , I want to automate the SAS em project. Is that possible to automate the project either by batch job or by scheduler.(every month with new history table/input table and new score table )
Please help me with this .
Thanks,
Sathya .