giuseppec / giuseppec/iml

FeatureImp + mlr3 Learner that predicts probabilities does not work

Open
#134 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
R
Stars
503
Forks
87
PR merge metrics
No merged PRs in 30d

Description

If I want to compute the importance for a measure based on probabilities (e.g., brier score), the `FeatureImp` is never calculated on the probabilities, even if I manually use a `predict.function`:

```r
library("mlr3")
library("iml")
credit.task = tsk("german_credit")
lrn = lrn("classif.rpart", predict_type = "prob")
model = lrn$train(credit.task)
data = credit.task$data()

# write a measure that just prints the `predicted` that will be used to calculate the measure
measure_print_predicted = function(actual, predicted) {
cat(head(predicted)) # have a look at how predicted looks like
}

pred = Predictor$new(model, data = data, y = "credit_risk")
imp = FeatureImp$new(pred, loss = measure_print_predicted, n.repetitions = 1)
# 1 2 1 2 2 1
```
It seems that internally the class is converted as numeric values (1 and 2), which makes it impossible to compute measures based on probabilities. I then tried to directly use a manually written `predict.function` which also did not work:

```r
# use a manually written predict function that returns probabilities
predict_good_prob = function(model, newdata) predict(model, newdata, predict_type = "prob")[, "good"]
head(predict_good_prob(model, data))

# use this predict function for IML method
pred = Predictor$new(model, data = data, y = "credit_risk", predict.function = predict_good_prob)
imp = FeatureImp$new(pred, loss = measure_print_predicted, n.repetitions = 1)
# 1 2 1 2 2 1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.