giuseppec / giuseppec/iml

iml complains about future.globals.maxSize even when no future plan is set

Open
#220 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
503
Forks
87
PR merge metrics
No merged PRs in 30d

Description

It seems as though {iml} always uses `future.apply` functions even when no `future::plan()` has been set, as you can see here:

```r
library(iml)
library(randomForest)

data("Boston", package = "MASS")

rf <- randomForest(medv ~ ., data = Boston, n.trees = 10)
X <- Boston[which(names(Boston) != "medv")]
predictor <- Predictor$new(rf, data = X, y = Boston$medv)

options(future.globals.maxSize = 1)

FeatureImp$new(predictor, loss = "mape")

# Error in `getGlobalsAndPackages()`:
# ! The total size of the 2 globals exported is 56.59 MiB.. This exceeds the maximum allowed size of 1 bytes (option 'future.globals.maxSize'). There are two globals: ‘FUN’ (28.30 MiB of class ‘function’) and ‘MoreArgs’ (28.29 MiB of class ‘list’)
```

I discovered this after retraining a model whose training data had grown, pushing it just above the default `future.globals.maxSize=500` MB. I was really confused at first because I explicitly chose not to configure a `future::plan` because I have a `predict.function` that handles parallelization.

I think {iml} should have a more explicit way of enabling parallelization to avoid confusion. It would be a better developer experience to not need to read the parallelization vignette in order to debug the default single-threaded behavior of the package.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.