topic Re: Weight in Gradient Boosting in SAS Data Science

Weight in Gradient Boosting

Babloo — Wed, 06 Jul 2016 15:34:47 GMT

What is weight in Gradient Boosting? Appreciate if someone point to me to some simple example of Gradient Boosting.

Thanks!

Re: Weight in Gradient Boosting

rayIII — Wed, 06 Jul 2016 15:53:22 GMT

This may help.

https://support.sas.com/resources/papers/proceedings14/SAS133-2014.pdf

Ray

Re: Weight in Gradient Boosting

Babloo — Thu, 07 Jul 2016 11:26:35 GMT

Thanks for the document, but still i don't understand the term 'weight'.

Re: Weight in Gradient Boosting

PadraicGNeville — Thu, 07 Jul 2016 15:04:44 GMT

"weight" could refer to weighting the observations or to weighting the trees in the model.

A boosting model typically consists of a sum of decision trees trained sequentially. Some algorithms describe the sum as weighted.

In Adaboost, the original boosting algorithm, observations are given weights before training a tree. The weights are different for each tree.

In gradient boosting algorithms do not use weights like this. Instead, the algorithm modifies the target values input to a tree.

The EM Boosting node uses gradient boosting. In some rare occassions, people assign weighted values to the observations at the start in order to match proportions of groups in the training data with those in a future population to which the model will be applied.

Re: Weight in Gradient Boosting

Babloo — Thu, 07 Jul 2016 18:57:08 GMT

Thanks for the reply, however I still don't understand the meaning of 'weighting the observations or to weighting the trees'. Appreciate if you tell me in layman's terms

Re: Weight in Gradient Boosting

PadraicGNeville — Thu, 07 Jul 2016 19:30:35 GMT

A weight is a positive number.

"Weighting observations" means a positive number is associated with each observation, and the algorithm utilizes that number somehow. Intuitively, observations with larger weights influence the algorithm more than observations with smaller weights. When Adaboost has created 10 trees in its boosting model, it will assign small weights to observations it is predicting well, so that Adaboost will create the 11th tree focusing on observations it hitherto predicted poorly.

"Weighting trees" means the predictions from the trees are multiplied by a weight: P(X) = W1 T1(X) + W2 T2(X),

where Ti(X) is the prediction of tree i for inputs X, and Wi are the weights. Sometimes the gradient boosting algorith is explained: first train the next tree (T2), and then find a single number (W2) that works best.