giuseppec / giuseppec/iml

Class FeatureImp$new() in multiclass problem computes metrics on probabilities instead of class labels

Open
#223 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
503
Forks
87
PR merge metrics
No merged PRs in 30d

Description

When computing Permutation Feature Importance (PFI) using `FeatureImp$new()` _per class_ in a multiclass classification task, the loss function appears to be evaluated directly on predicted probabilities instead of class labels, producing incorrect values for metrics that expect lables rather than probabilities (e.g. `Metrics::ce`).

In contrast, when computing the global multiclass PFI, the probabilities are correctly converted to the label with maximum probability using `iml:::probs.to.labels()` before the loss function is applied. However, conversions are not applied when `Predictor$predict()` returns a single column of probabilities (as in class PFI).

```r
library(iml)
library(mlr3)
library(mlr3learners)
library(Metrics)

ce_fun <- function(actual, predicted) { # to explor the metric computation
cat("Actual: ", actual, "\n\n")
cat("Predicted: ", predicted, "\n\n")
cat("METRIC: ", Metrics::ce(actual, predicted), "\n\n")

return(Metrics::ce(actual, predicted))
}

tsk = as_task_classif(iris, target = "Species")
lrn = lrn("classif.ranger", predict_type = "prob")

lrn$train(tsk)

# Class Feature Importance ----

set.seed(123)
rows = sample(nrow(iris), 10)

mod_class <- Predictor$new(lrn,
data = iris[rows,], y = iris$Species[rows] == "setosa",
type = "prob", class = "setosa")

mod_class$predict(tsk$data())
iml:::probs.to.labels(mod_class$predict(tsk$data()))

imp <- FeatureImp$new(mod_class, loss = ce_fun, compare = "ratio", n.repetitions = 1)

```
Example output:

```
Actual: TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE

Predicted: 1 0 0.9995 0 0 1 0 0 0 0

METRIC: 0.1
```
A probability of `0.9995` to be `TRUE` is treated as a misclassification because the loss function receives a numeric probability and not a class label. Shouldn't it be converted to 1 (i.e., TRUE) using a threshold of 0.5? Or am I wrong?

```r
# For a more clear explanation (predicting on the first 10 rows, i.e, always setosa)

set.seed(123)
mod <- Predictor$new(lrn,
data = iris[1:10,], y = iris$Species[1:10] == "setosa",
type = "prob", class = "setosa")

mod$predict(tsk$data())
iml:::probs.to.labels(mod$predict(tsk$data()))

imp <- FeatureImp$new(mod, loss = ce_fun, compare = "ratio", n.repetitions = 1)

# Example output:
# Actual: TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
#
# Predicted: 1 1 1 1 1 1 1 0.9993333 0.9993333 1
#
# METRIC: 0.2
#
# The metric should not be 0?
```
```r
> sessionInfo()
R version 4.5.1 (2025-06-13 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26100)

Matrix products: default
LAPACK version 3.12.1

locale:
[1] LC_COLLATE=Italian_Italy.utf8 LC_CTYPE=Italian_Italy.utf8 LC_MONETARY=Italian_Italy.utf8 LC_NUMERIC=C
[5] LC_TIME=Italian_Italy.utf8

time zone: Europe/Rome
tzcode source: internal

attached base packages:
[1] tools grid parallel compiler stats graphics grDevices utils datasets methods base

other attached packages:
[1] Formula_1.2-5 labeling_0.4.3 withr_3.0.2 lattice_0.22-7 Rcpp_1.1.0 ranger_0.17.0
[7] Matrix_1.7-3 pkgconfig_2.0.3 parallelly_1.45.1 codetools_0.2-20 listenv_0.9.1 farver_2.1.2
[13] data.table_1.17.8 glue_1.8.0 vctrs_0.6.5 lifecycle_1.0.4 rstudioapi_0.17.1 digest_0.6.37
[19] magrittr_2.0.4 cli_3.6.5 mlr3misc_0.19.0 lgr_0.5.0 rlang_1.1.6 RColorBrewer_1.1-3
[25] pillar_1.11.1 paradox_1.0.1 Metrics_0.1.4 tibble_3.3.0 checkmate_2.3.3 backports_1.5.0
[31] palmerpenguins_0.1.1 generics_0.1.4 R6_2.6.1 ggplot2_3.5.2 uuid_1.2-1 scales_1.4.0
[37] globals_0.18.0 dichromat_2.0-0.1 tidyselect_1.2.1 rpart_4.1.24 crayon_1.5.3 dplyr_1.1.4
[43] gtable_0.3.6 future.apply_1.20.0 future_1.67.0 mlr3learners_0.13.0 mlr3_1.2.0 iml_0.11.4
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.