Model comparison in SAS VS How does this work

[PaalNavestad]

Hi, I'm using the model comparison in SAS/VS. It does some strange things. I have an RF model with reported Validation ASE of 197,611. I have a Gradient Boosting (DSG) model with a reported Validation ASE of 199,686. After running this through model comparison I get a validation ASE of 196,927 for the RF model and 202,275 for the DSG model. These numbers vary with the number of models being in the Model Comparison.Random forest variable importanceGradientBoosting variable importanceModel comparison HeatmapModelcomparison table

Of a larger concern is that the Variable importance is completely changed. The variable that in the DSG model is the second most important has suddenly become the 7 most important after Model comparison. Also the variable importance HeatMap is inconsistent with the variable importance table.