FeatureImp when add junk variable
- Dominant language
- R
- Stars
- 503
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
The iml manual describes FeatureImp as:
FeatureImp computes feature importance for prediction models. The importance is measured as the factor by which the model’s prediction error increases when the feature is shuffled.
However, when I estimate a linear model using lm and them add a junk variable to the data frame, FeatureImp gives a non-zero value for the importance of the junk variable. That contradicts the description because the junk variable has coefficient = 0 in the model, so shuffling it should have no effect.
If the shuffling algorithm also somehow involves taking subsets of the data (via n.repetitions??), then non-zero values for the junk variable could result. I note the n.repetitions = 1 and n.repetitions = number of observations gives feature importance == 0.
Can you clarify?
library(iml)
library(dplyr)
# parameters
set.seed(1234552)
nobs <- 100
sigma_sq3 <- 5
beta0 <- 5
beta1 <- -1
beta2 <- 0.5
# generate data
df <- as.data.frame(rnorm(nobs, 0, sqrt(sigma_sq3))) # normal errors
names(df) <- c("e")
df <- df %>%
mutate(x1 = runif(nobs, -3, 3),
x2 = 0.25*x1 + rnorm(nobs, 0, 1),
y3 = beta0 + beta1*x1 + beta2*x2 + e)
lm3 <- lm(y3 ~ x1 + x2, data = df)
print(summary(lm3))
set.seed(232452)
data <- df[, c("y3", "x1", "x2")]
mod <- Predictor$new(lm3, y = "y3", data = data)
imp <- FeatureImp$new(mod, loss="mse", compare = "difference", n.repetitions = 10)
plot(imp)
print(imp)
# add junk variable to data, but not to model
set.seed(232452)
data$junk <- rnorm(nrow(data), 0, 1)
mod <- Predictor$new(lm3, y = "y3", data = data)
imp <- FeatureImp$new(mod, loss="mse", compare = "difference", n.repetitions = 10)
plot(imp)
print(imp)
Contributor guide
Assessment
This issue has not been assessed yet.