Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Oversampling in Enterprise Miner with a rare event fixed proportion

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 11
Accepted Solution

Oversampling in Enterprise Miner with a rare event fixed proportion

Hi,

I need to oversample in Enterprise Miner with a rare event fixed proportion.

Starting from a database with a 0.5% of rare event, I need to specify the proportion of rare event in the sample. 50:50 is not ok, because database is too small in this case.

Thank you very much.

F


Accepted Solutions
Solution
‎10-02-2014 12:49 PM
SAS Employee
Posts: 1

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio, use the "Level Based" options in the Sampling node as demonstrated in the video that Miguel mentioned. You can set any sampling proportions that you want.

Also see the section on "Detecting Rare Classes" in the documentation for Enterprise Miner.

View solution in original post


All Replies
Super Contributor
Posts: 336

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Hi Fabio,

This usage note details the steps on how to model a rare event using oversampling.

Link here: 24205 - Rare event oversampling for model fitting in SAS® Enterprise Miner(tm)

It includes a video too!

Thanks,

Miguel

Occasional Contributor
Posts: 11

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Thank you, Miguel.

The note help to create a 50-50 sample. But the proportion of rare event is to big for me in this sample. I need a sample with a 10% of rare event proportion.

Is it possible?

Solution
‎10-02-2014 12:49 PM
SAS Employee
Posts: 1

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio, use the "Level Based" options in the Sampling node as demonstrated in the video that Miguel mentioned. You can set any sampling proportions that you want.

Also see the section on "Detecting Rare Classes" in the documentation for Enterprise Miner.

Occasional Contributor
Posts: 11

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Thank you everybody for your precious help

Super Contributor
Posts: 336

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio,

I added a detailed example this morning on how to adjust probabilities for a 50/50 oversample using a Decisions node.

Take a look here:.

I hope it helps,

Miguel

N/A
Posts: 1

Re: Oversampling in Enterprise Miner with a rare event fixed proportion

Fabio

I am not sure if it will be of some help, but you may take a look at this paper:

http://gking.harvard.edu/files/0s.pdf

The authors studies the opposite of the problem you are trying to find a solution. They suggest an adjustment procedure for the data set with huge number observations but relatively few events. Nevertheless, you may be able to implement the same sampling design principle in your data set and estimation procedure as the authors suggest. I have used their methodology for large data sets with relatively low number of cases, and it is a quite effective and slick approach.   

Good luck.

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 1459 views
  • 7 likes
  • 4 in conversation