giuseppec / giuseppec/featureImportance

Cannot reproduce measure (delta L) values of featureImportance for individual observations outside of featureImportance

Open
#6 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
33
Forks
9
PR merge metrics
No merged PRs in 30d

Description

I would like to get the local feature importance of the ith observation for individual variables as described in Casalicchio et al. section 4 lines 7-8 by integrating the ICI curve for the ith observation however, I am unsure how to interpret the negative delta L values reported by the function and cannot reproduce them outside of the featureImportance() function. Could this be an issue with not using mlr package? Or am I not interpreting the delta L value correctly?

Below I set up a gbm model like in this issue: https://github.com/giuseppec/featureImportance/issues/5

```
library(DALEX)
library(featureImportance)
library(gbm)
library(pracma)
library(tidyverse)

mod <- gbm(m2.price~., data = apartments)
summary(mod)

pred.fun <- function(object, newdata){
preds <- predict(mod, newdata, n.trees = 100)
}

# you need to define the performance measure, I use the rmse here
measure <- function(truth, response)
sqrt(mean((truth-response)^2))

# calculate the local permutation feature importance for all features
imp <- featureImportance(mod, data = apartments, target = "m2.price", n.feat.perm = 100,
local = TRUE, predict.fun = pred.fun, measures = list(rmse=measure))

# save permutation importance to a data frame
imp_df <- imp$importance

# get the values for the surface feature only
surface <- filter(imp_df, features == "surface"
# get the values for the surface feature only for the 23rd observation, which has negative reported measures (change in rmse?)
surface_23 <- filter(surface, row.id == 23) %>% mutate(feature.value = as.numeric(as.character(feature.value))) %>% arrange(feature.value) # need to convert feature.value to numeric
# change in rmse for observation 23
ggplot(surface_23, aes(x = feature.value, y = rmse)) + geom_line() # I would like to integrate this curve to get a summary measure of the local feature importance for this observation

# try manually calculating the rmse and change in rmse for a prediction
apartments[23,]
predict(mod, apartments[23,], n.trees = 100)
rmse(sim = 4776.9, obs = 5170)
new <- apartments[23,]
new$surface <- 50 # replace the surface variable value with the permuted value for permutation 66
predict(mod, new, n.trees = 100)
rmse(sim = 5624.19, obs = 5170)
delL <- 393.1-454.19 # this should be the value described in Casalicchio et al. section 4 lines 1-6 correct?
delL # but feature importance function reports -44.26 for this case
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.