Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Weight in Gradient Boosting

Accepted Solution Solved
Reply
Super Contributor
Posts: 426
Accepted Solution

Weight in Gradient Boosting

What is weight in Gradient Boosting? Appreciate if someone point to me to some simple example of Gradient Boosting.

 

Thanks!


Accepted Solutions
Solution
‎07-08-2016 08:25 AM
SAS Employee
Posts: 32

Re: Weight in Gradient Boosting

A weight is a positive number.

 

"Weighting observations" means a positive number is associated with each observation, and the algorithm utilizes that number somehow.  Intuitively, observations with larger weights influence the algorithm more than observations with smaller weights.  When  Adaboost has created 10 trees in its boosting model, it will assign small weights to observations it is predicting well, so that Adaboost will create the 11th tree focusing on observations it hitherto predicted poorly.

 

"Weighting trees" means the predictions from the trees are multiplied by a weight:  P(X) = W1 T1(X) + W2 T2(X),

where Ti(X) is the prediction of tree i for inputs X, and Wi are the weights.  Sometimes the gradient boosting algorith is explained: first train the next tree (T2), and then find a single number (W2) that works best.

View solution in original post


All Replies
SAS Employee
Posts: 106

Re: Weight in Gradient Boosting

Super Contributor
Posts: 426

Re: Weight in Gradient Boosting

Thanks for the document, but still i don't understand the term 'weight'.

SAS Employee
Posts: 32

Re: Weight in Gradient Boosting

"weight" could refer to weighting the observations or to weighting the trees in the model. 

 

A boosting model typically consists of a sum of decision trees trained sequentially.  Some algorithms describe the sum as weighted.

In Adaboost, the original boosting algorithm, observations are given weights before training a tree.   The weights are different for each tree.

 

In gradient boosting algorithms do not use weights like this.  Instead, the algorithm modifies the target values input to a tree.

 

The EM Boosting node uses gradient boosting.  In some rare occassions, people assign weighted values to the observations at the start in order to match proportions of groups in the training data  with those in a future population to which the model will be applied.

Super Contributor
Posts: 426

Re: Weight in Gradient Boosting

Thanks for the reply, however I still don't understand the meaning of 'weighting the observations or to weighting the trees'. Appreciate if you tell me in layman's terms

Solution
‎07-08-2016 08:25 AM
SAS Employee
Posts: 32

Re: Weight in Gradient Boosting

A weight is a positive number.

 

"Weighting observations" means a positive number is associated with each observation, and the algorithm utilizes that number somehow.  Intuitively, observations with larger weights influence the algorithm more than observations with smaller weights.  When  Adaboost has created 10 trees in its boosting model, it will assign small weights to observations it is predicting well, so that Adaboost will create the 11th tree focusing on observations it hitherto predicted poorly.

 

"Weighting trees" means the predictions from the trees are multiplied by a weight:  P(X) = W1 T1(X) + W2 T2(X),

where Ti(X) is the prediction of tree i for inputs X, and Wi are the weights.  Sometimes the gradient boosting algorith is explained: first train the next tree (T2), and then find a single number (W2) that works best.

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 666 views
  • 0 likes
  • 3 in conversation