giuseppec / giuseppec/featureImportance
Cannot reproduce measure (delta L) values of featureImportance for individual observations outside of featureImportance
- Dominant language
- R
- Stars
- 33
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
I would like to get the local feature importance of the ith observation for individual variables as described in Casalicchio et al. section 4 lines 7-8 by integrating the ICI curve for the ith observation however, I am unsure how to interpret the negative delta L values reported by the function and cannot reproduce them outside of the featureImportance() function. Could this be an issue with not using mlr package? Or am I not interpreting the delta L value correctly?
Below I set up a gbm model like in this issue: https://github.com/giuseppec/featureImportance/issues/5
```
library(DALEX)
library(featureImportance)
library(gbm)
library(pracma)
library(tidyverse)
mod <- gbm(m2.price~., data = apartments)
summary(mod)
pred.fun <- function(object, newdata){
preds <- predict(mod, newdata, n.trees = 100)
}
# you need to define the performance measure, I use the rmse here
measure <- function(truth, response)
sqrt(mean((truth-response)^2))
# calculate the local permutation feature importance for all features
imp <- featureImportance(mod, data = apartments, target = "m2.price", n.feat.perm = 100,
local = TRUE, predict.fun = pred.fun, measures = list(rmse=measure))
# save permutation importance to a data frame
imp_df <- imp$importance
# get the values for the surface feature only
surface <- filter(imp_df, features == "surface"
# get the values for the surface feature only for the 23rd observation, which has negative reported measures (change in rmse?)
surface_23 <- filter(surface, row.id == 23) %>% mutate(feature.value = as.numeric(as.character(feature.value))) %>% arrange(feature.value) # need to convert feature.value to numeric
# change in rmse for observation 23
ggplot(surface_23, aes(x = feature.value, y = rmse)) + geom_line() # I would like to integrate this curve to get a summary measure of the local feature importance for this observation
# try manually calculating the rmse and change in rmse for a prediction
apartments[23,]
predict(mod, apartments[23,], n.trees = 100)
rmse(sim = 4776.9, obs = 5170)
new <- apartments[23,]
new$surface <- 50 # replace the surface variable value with the permuted value for permutation 66
predict(mod, new, n.trees = 100)
rmse(sim = 5624.19, obs = 5170)
delL <- 393.1-454.19 # this should be the value described in Casalicchio et al. section 4 lines 1-6 correct?
delL # but feature importance function reports -44.26 for this case
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.