FeatureImp + mlr3 Learner that predicts probabilities does not work
- Dominant language
- R
- Stars
- 503
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
If I want to compute the importance for a measure based on probabilities (e.g., brier score), the `FeatureImp` is never calculated on the probabilities, even if I manually use a `predict.function`:
```r
library("mlr3")
library("iml")
credit.task = tsk("german_credit")
lrn = lrn("classif.rpart", predict_type = "prob")
model = lrn$train(credit.task)
data = credit.task$data()
# write a measure that just prints the `predicted` that will be used to calculate the measure
measure_print_predicted = function(actual, predicted) {
cat(head(predicted)) # have a look at how predicted looks like
}
pred = Predictor$new(model, data = data, y = "credit_risk")
imp = FeatureImp$new(pred, loss = measure_print_predicted, n.repetitions = 1)
# 1 2 1 2 2 1
```
It seems that internally the class is converted as numeric values (1 and 2), which makes it impossible to compute measures based on probabilities. I then tried to directly use a manually written `predict.function` which also did not work:
```r
# use a manually written predict function that returns probabilities
predict_good_prob = function(model, newdata) predict(model, newdata, predict_type = "prob")[, "good"]
head(predict_good_prob(model, data))
# use this predict function for IML method
pred = Predictor$new(model, data = data, y = "credit_risk", predict.function = predict_good_prob)
imp = FeatureImp$new(pred, loss = measure_print_predicted, n.repetitions = 1)
# 1 2 1 2 2 1
```
Contributor guide
Assessment
This issue has not been assessed yet.