topic Re: How is variable importance calculated for a gradient boosting node in e-miner? in SAS Data Science

How is variable importance calculated for a gradient boosting node in e-miner?

PadmaroopaK — Fri, 11 Sep 2020 13:45:33 GMT

I have a simple gradient boosting model (maximum branch = 2 , maximum depth = 1 {Adaboost} ) in e-miner(v 14.1) with binary target and mostly interval inputs(~500 variables). I will be choosing variables if the variable importance > 0.05 for both training and validation datasets. However, I am trying to understand the mathematics behind how the "variable importance" is calculated. I read the documentation (decision tree variable importance ) but its very vague. I was wondering if anyone could shed light on how it is calculated with a simple example? It will be very helpful.

Re: How is variable importance calculated for a gradient boosting node in e-miner?

pink_poodle — Sat, 19 Sep 2020 00:11:56 GMT

feature importance for a single decision tree - the amount that each attribute split point improves the performance measure, weighted by the number of observations the node is responsible for. The performance measure may be the purity (Gini index) used to select the split points, or another more specific error function.
overall feature importance - feature importances averaged across all of the the decision trees within the model.

Re: How is variable importance calculated for a gradient boosting node in e-miner?

gcjfernandez — Fri, 02 Oct 2020 06:10:22 GMT

The Gradient Boosting node in SAS EM provides two approaches to evaluating the importance of a variable: split-based and observation-based.
The split-based approach uses the reduction in the sum of squares from splitting a node, summing over all nodes.
The observation-based approach uses the increase in a fit statistic due to seeing values of a variable uninformative.
Measures of variable importance generally underestimate the importance of correlated variables.

Two correlated variables could make a similar contribution to a model. The total contribution is usually divided between them, and neither variable acquires the rank it deserves.

Eliminating either variable generally increases the contribution attributed to the other.

Re: How is variable importance calculated for a gradient boosting node in e-miner?

PadmaroopaK — Mon, 05 Oct 2020 14:09:43 GMT

Thank you for your response.

I am looking at the split-based approach in my model. I find that reduction in sum of squares from the splitting node explanation a little abstract. Is there any SAS white paper or any way to see that actual calculation for atleast one variable? I am interested in seeing that back end calculation that produces those numbers.

Thanks!

Re: How is variable importance calculated for a gradient boosting node in e-miner?

PadmaroopaK — Mon, 05 Oct 2020 14:11:07 GMT

Is there a way for me to see this back end computation in e-miner?

Re: How is variable importance calculated for a gradient boosting node in e-miner?

pink_poodle — Mon, 05 Oct 2020 14:24:29 GMT

Here are some formulas and background:
https://documentation.sas.com/?docsetId=casml&docsetTarget=viyaml_treesplit_details02.htm&docsetVersion=3.0&locale=en

Re: How is variable importance calculated for a gradient boosting node in e-miner?

gcjfernandez — Mon, 05 Oct 2020 14:42:21 GMT

I am attaching the screenshot from SAS Enterprise miner Reference documentation 14.3 where you can find the official computation description.