FeatureImp returns same importance value for every variable with mlr3 BART model
- Dominant language
- R
- Stars
- 503
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
I am training several ML models using the mlr3 package and have been using iml to retrieve permutation importance for variables in my data. However, I have noticed that, for BART models, the variable importance is exactly the same for _every_ variable. Below is code to reproduce this issue using a toy dataset. Even the variable which is completely unrelated to the outcome variable has the same variable importance as the others.
```
library(mlr3verse)
mlr3extralearners::install_learners("regr.bart")
#library(iml)
n <- 100
set.seed(123)
x1 <- rnorm(n, 4, 5)
x2 <- sample(c("a", "b","c"), size = n, replace = T)
x3 <- sample(letters[1:4], size = n, replace = T)
x4_noise <- rnorm(n, 1, 6)
y <- 3 + 2*x1 + 5*(x2=="a") - 10*(x2=="b") + 25*(x2=="c") +
4*(x3=="a") - 4*(x3=="b") + 5*(x3=="c") +10*(x3=="d") -
50*(x3=="d")*(x2=="b") +
rnorm(n, 0, 3)
df <- data.frame(x1 = x1, x2 = factor(x2),
x3 = factor(x3), x4_noise = x4_noise,
y = y)
task <- as_task_regr(df, target = "y")
gr <- po("scale") %>>% po("encode") %>>% lrn("regr.bart")
grl <- GraphLearner$new(gr)
grl$train(task)
model <- iml::Predictor$new(grl, data = df, y = "y")
imp_mod <- iml::FeatureImp$new(model, loss = "rmse",
n.repetitions = 50,
compare = "ratio")
imp_mod$results
```
Output:
```
feature importance.05 importance importance.95 permutation.error
1 x1 15.61669 15.61669 15.61669 24.96426
2 x2 15.61669 15.61669 15.61669 24.96426
3 x3 15.61669 15.61669 15.61669 24.96426
4 x4_noise 15.61669 15.61669 15.61669 24.96426
```
Contributor guide
Research direction
Start by running the R reproducer with mlr3verse, mlr3extralearners, iml::Predictor$new, and iml::FeatureImp$new as shown. Trace the FeatureImp and Predictor entry points to determine why the BART GraphLearner produces identical permutation results for every feature. Done means the reproducer distinguishes feature importance values, including the unrelated x4_noise variable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100