topic For the extremely unbalanced data (less than 1% event incidence), what's the best method to deal? in SAS Data Science

For the extremely unbalanced data (less than 1% event incidence), what's the best method to deal?

Jade_SAS — Thu, 16 May 2019 18:46:54 GMT

Dear all,

I am working on one project with extremely unbalanced data which has less than 1% event incidence (800 out of total 114600 obs), what's the best method to deal with this kind of data? As the expecting goal of this project is to provide rules to distinguish the bad event (less than 1%) in the future, I am using decision tree right now. But the performance is really bad, the decision tree will not go further, only stay with the root node. Any suggestion on dealing with this kind of problem is welcome!!!

BTW: I am using SAS miner 14.2 on the SAS Linux.

Thank you!

Jade

Re: For the extremely unbalanced data (less than 1% event incidence), what's the best method to deal

Reeza — Thu, 16 May 2019 18:55:54 GMT

Without any more information, my suggestion would be to switch to a case control option where you take your 800 and match it 1:N against the big data set, with N controls per case.

I would then bootstrap it to try that approach X times and see if my model is stable. I have no idea how to do this in EM and don't have access to it though. Technically could be done in Base SAS though.

Re: For the extremely unbalanced data (less than 1% event incidence), what's the best method to deal

Jade_SAS — Fri, 24 May 2019 17:24:18 GMT

Thank you Reeza!

Is there a reference paper of this procedure? Thank you!

Jade

Re: For the extremely unbalanced data (less than 1% event incidence), what's the best method to deal

Reeza — Fri, 24 May 2019 17:29:44 GMT

Not really, you can look at PROC PSMATCH to get the matches and then PROC PHREG for conditional logistic regression. The documentation has examples for each.

Re: For the extremely unbalanced data (less than 1% event incidence), what's the best method to deal

Jade_SAS — Fri, 24 May 2019 17:39:23 GMT

Thank you Reeza!