topic How to adjust probability predicted using logistic regression after oversampling in SAS Data Science

How to adjust probability predicted using logistic regression after oversampling

Saurabh_Rana — Tue, 28 Dec 2021 11:13:53 GMT

I have created a logistic regression model using the E-Miner tool where event probability in population base was 0.06, after oversampling I created a base where event probability is 0.2. Now how can I adjust the probabilities according to the population base using SAS code in Enterprise Guide?

I found below mentioned formula in another post from the SAS community-how-to-adjust-probabilities-after-oversampling

P_i** = ( P_i* x R_0 x P_1) / ( (1-P_i*) (R_1)(P_0) + (P_i*)(R_0)(P_1) )

where:

P_i* is the unadjusted probability you get from your model

R_0 and R_1 are the sample proportions of 1 and 0 respectively

P_0 and P_1 are the original event and non_event rates (population rates)

P_i** is the true probability

But using this formula I am getting adjusted probability to be higher than actual probability.

Re: How to adjust probability predicted using logistic regression after oversampling

Ksharp — Tue, 28 Dec 2021 12:16:52 GMT

Check SCORE statment of PROC LOGISTIC :

Re: How to adjust probability predicted using logistic regression after oversampling

sbxkoenk — Tue, 28 Dec 2021 12:53:58 GMT

Hello,

First of all : Make sure you are not mixing up the (#2) target levels.

And check what the target level is that you are predicting : is you model giving probabilities for level_1 or for level_2?

If the above is OK, here's how you adjust :

Usage Note 22601: Adjusting for oversampling the event level in a binary logistic model
https://support.sas.com/kb/22/601.html

And from the Enterprise Miner documentation :
SAS® Enterprise Miner™ 15.1 Extension Nodes: Developer’s Guide
https://go.documentation.sas.com/doc/en/emxndg/15.1/p1vqpbjwoo4bv7n1sw77e0z64xxs.htm

Cheers,

Koen

Re: How to adjust probability predicted using logistic regression after oversampling

Saurabh_Rana — Tue, 28 Dec 2021 18:08:12 GMT

Let me show you the calculation for an example where Predicted Probability is 0.6:-

Predicted Probability=0.6

Sample Event Proportion=0.2

Sample Non Event Proportion=0.8

Population Event Proportion=0.06

Population Non Event Proportion=0.94

Adjusted Probability=(0.6*0.2*0.94)/[(0.4*0.8*0.06)+(0.6*0.2*0.94)] = 0.8545

Adjusted Probability( 0.8545 ) > Predicted Probability(0.6)

Please tell me where I am making a mistake while calculating adjusted probability?

Re: How to adjust probability predicted using logistic regression after oversampling

sbxkoenk — Tue, 28 Dec 2021 22:22:12 GMT

data _NULL_;
*Predicted Probability=0.6              ; *OldPost(i,t);
*Sample Event Proportion=0.2            ; *OldPrior(t) ;
*Sample Non Event Proportion=0.8        ; 
*Population Event Proportion=0.06       ; *Prior(t)    ;
*Population Non Event Proportion=0.94   ; 

Post_i_t = (0.6 * 0.06 / 0.2) / ( (0.6 * 0.06 / 0.2) + (0.4 * 0.94 / 0.8) );
put Post_i_t=;
run;

Post_i_t = 0.2769230769

Cheers,

Koen

Re: How to adjust probability predicted using logistic regression after oversampling

Saurabh_Rana — Wed, 29 Dec 2021 06:56:50 GMT

Using this method adjusting probability is coming out to be lower than predicted probability but when I take the mean of all the adjusted probabilities it is not coming out to be 0.06, ideally, the mean of adjusted probability should be equal to event probability in the population base. Like if I take the mean of predicted probability it is coming out to be 0.2 as expected which is the event probability in the sample base.

Re: How to adjust probability predicted using logistic regression after oversampling

sbxkoenk — Wed, 29 Dec 2021 17:24:57 GMT

Hello,

You are right in expecting the mean of all adjusted probabilities to be (approximately) the event rate in the population base.

I do not know why that's not the case with your data.

However, why are you adjusting these probabilities "manually"?

If you use the Enterprise Miner target profiler, then the correct posterior probabilities (adjusted for the real priors) are automatically returned by the software.

See here :

SAS® Enterprise Miner™ 15.1: Reference Help

Predictive Modeling : https://go.documentation.sas.com/doc/en/emref/15.1/p0qiq0a4vnebuzn16v8fzossk4gp.htm

Enterprise Miner Target Profiler : https://go.documentation.sas.com/doc/en/emref/15.1/n0z1mtvsscypjqn1ediv223jq5iy.htm

Kind regards,

Koen

Re: How to adjust probability predicted using logistic regression after oversampling

Saurabh_Rana — Sun, 02 Jan 2022 20:02:11 GMT

I will try to adjust probability using the E-Miner method but is there no mathematical formula through which I could adjust probability?

Re: How to adjust probability predicted using logistic regression after oversampling

sbxkoenk — Sun, 02 Jan 2022 20:26:50 GMT

Hello,

The mathematical formula is in one of my posts above.

(and in the doc : SAS® Enterprise Miner™ 15.1 Extension Nodes: Developer’s Guide
https://go.documentation.sas.com/doc/en/emxndg/15.1/p1vqpbjwoo4bv7n1sw77e0z64xxs.htm).

The formula is also here (marked as solution).
Subject : Urgent, how to adjust probabilities after oversampling? Please Help, Thank you

https://communities.sas.com/t5/SAS-Data-Mining-and-Machine/Urgent-how-to-adjust-probabilities-after-oversampling-Please/td-p/170809

I have successfully done it the formula-way myself several times, but cannot locate these programs any more.

Good luck,

Koen