giuseppec / giuseppec/iml

FeatureImp returns same importance value for every variable with mlr3 BART model

Open
#202 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
503
Forks
87
PR merge metrics
No merged PRs in 30d

Description

I am training several ML models using the mlr3 package and have been using iml to retrieve permutation importance for variables in my data. However, I have noticed that, for BART models, the variable importance is exactly the same for _every_ variable. Below is code to reproduce this issue using a toy dataset. Even the variable which is completely unrelated to the outcome variable has the same variable importance as the others.

```
library(mlr3verse)
mlr3extralearners::install_learners("regr.bart")
#library(iml)
n <- 100
set.seed(123)
x1 <- rnorm(n, 4, 5)
x2 <- sample(c("a", "b","c"), size = n, replace = T)
x3 <- sample(letters[1:4], size = n, replace = T)
x4_noise <- rnorm(n, 1, 6)

y <- 3 + 2*x1 + 5*(x2=="a") - 10*(x2=="b") + 25*(x2=="c") +
4*(x3=="a") - 4*(x3=="b") + 5*(x3=="c") +10*(x3=="d") -
50*(x3=="d")*(x2=="b") +
rnorm(n, 0, 3)

df <- data.frame(x1 = x1, x2 = factor(x2),
x3 = factor(x3), x4_noise = x4_noise,
y = y)
task <- as_task_regr(df, target = "y")
gr <- po("scale") %>>% po("encode") %>>% lrn("regr.bart")
grl <- GraphLearner$new(gr)
grl$train(task)

model <- iml::Predictor$new(grl, data = df, y = "y")
imp_mod <- iml::FeatureImp$new(model, loss = "rmse",
n.repetitions = 50,
compare = "ratio")
imp_mod$results
```

Output:
```
feature importance.05 importance importance.95 permutation.error
1 x1 15.61669 15.61669 15.61669 24.96426
2 x2 15.61669 15.61669 15.61669 24.96426
3 x3 15.61669 15.61669 15.61669 24.96426
4 x4_noise 15.61669 15.61669 15.61669 24.96426
```

Contributor guide

Open the contributing guide

Research direction

Start by running the R reproducer with mlr3verse, mlr3extralearners, iml::Predictor$new, and iml::FeatureImp$new as shown. Trace the FeatureImp and Predictor entry points to determine why the BART GraphLearner produces identical permutation results for every feature. Done means the reproducer distinguishes feature importance values, including the unrelated x4_noise variable.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.