giuseppec / giuseppec/iml

FeatureImp when add junk variable

Open
#151 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
503
Forks
87
PR merge metrics
No merged PRs in 30d

Description

The iml manual describes FeatureImp as:

FeatureImp computes feature importance for prediction models. The importance is measured as the factor by which the model’s prediction error increases when the feature is shuffled.

However, when I estimate a linear model using lm and them add a junk variable to the data frame, FeatureImp gives a non-zero value for the importance of the junk variable. That contradicts the description because the junk variable has coefficient = 0 in the model, so shuffling it should have no effect.

If the shuffling algorithm also somehow involves taking subsets of the data (via n.repetitions??), then non-zero values for the junk variable could result. I note the n.repetitions = 1 and n.repetitions = number of observations gives feature importance == 0.

Can you clarify?

library(iml)
library(dplyr)

# parameters
set.seed(1234552)
nobs <- 100
sigma_sq3 <- 5
beta0 <- 5
beta1 <- -1
beta2 <- 0.5

# generate data
df <- as.data.frame(rnorm(nobs, 0, sqrt(sigma_sq3))) # normal errors
names(df) <- c("e")
df <- df %>%
mutate(x1 = runif(nobs, -3, 3),
x2 = 0.25*x1 + rnorm(nobs, 0, 1),
y3 = beta0 + beta1*x1 + beta2*x2 + e)

lm3 <- lm(y3 ~ x1 + x2, data = df)
print(summary(lm3))

set.seed(232452)
data <- df[, c("y3", "x1", "x2")]
mod <- Predictor$new(lm3, y = "y3", data = data)
imp <- FeatureImp$new(mod, loss="mse", compare = "difference", n.repetitions = 10)
plot(imp)
print(imp)

# add junk variable to data, but not to model
set.seed(232452)
data$junk <- rnorm(nrow(data), 0, 1)
mod <- Predictor$new(lm3, y = "y3", data = data)
imp <- FeatureImp$new(mod, loss="mse", compare = "difference", n.repetitions = 10)
plot(imp)
print(imp)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.