topic Re: Oversampling in Enterprise Miner with a rare event fixed proportion in SAS Data Science

Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio — Fri, 26 Sep 2014 16:32:34 GMT

Hi,

I need to oversample in Enterprise Miner with a rare event fixed proportion.

Starting from a database with a 0.5% of rare event, I need to specify the proportion of rare event in the sample. 50:50 is not ok, because database is too small in this case.

Thank you very much.

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

M_Maldonado — Mon, 29 Sep 2014 15:10:29 GMT

Hi Fabio,

This usage note details the steps on how to model a rare event using oversampling.

Link here: 24205 - Rare event oversampling for model fitting in SAS® Enterprise Miner(tm)

It includes a video too!

Thanks,

Miguel

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio — Thu, 02 Oct 2014 07:50:28 GMT

Thank you, Miguel.

The note help to create a 50-50 sample. But the proportion of rare event is to big for me in this sample. I need a sample with a 10% of rare event proportion.

Is it possible?

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

jsienna — Thu, 02 Oct 2014 13:47:05 GMT

Fabio

I am not sure if it will be of some help, but you may take a look at this paper:

http://gking.harvard.edu/files/0s.pdf

The authors studies the opposite of the problem you are trying to find a solution. They suggest an adjustment procedure for the data set with huge number observations but relatively few events. Nevertheless, you may be able to implement the same sampling design principle in your data set and estimation procedure as the authors suggest. I have used their methodology for large data sets with relatively low number of cases, and it is a quite effective and slick approach.

Good luck.

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

WarrenSarle — Thu, 02 Oct 2014 16:49:14 GMT

Fabio, use the "Level Based" options in the Sampling node as demonstrated in the video that Miguel mentioned. You can set any sampling proportions that you want.

Also see the section on "Detecting Rare Classes" in the documentation for Enterprise Miner.

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio — Fri, 03 Oct 2014 15:49:26 GMT

Thank you everybody for your precious help

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

M_Maldonado — Fri, 03 Oct 2014 17:48:42 GMT

Fabio,

I added a detailed example this morning on how to adjust probabilities for a 50/50 oversample using a Decisions node.

Take a look here:.

I hope it helps,

Miguel