futureverse / futureverse/future.apply
PERFORMANCE: Search large `X`:s for globals is slow [SOLVED]
- Dominant language
- R
- Stars
- 218
- Forks
- 20
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 1
Description
In future.apply 1.0.0, `future_lapply(X, ...)` searches also `X` for possible globals (Issue #12). For long `X`:s this introduces a significant overhead, especially if `X` does _not_ contain any globals and we wouldn't have to search `X` in the first place. For example,
```r
X <- vector("list", length = 100e3)
y <- future_lapply(X, FUN = identity)
```
All the slowness comes from an internal:
```r
gp <- future::getGlobalsAndPackages(X, globals = TRUE)
```
Following the code, this is slow because
```r
names <- globals::findGlobals(X)
```
is slow, which in turn is because it effetively does:
```r
names <- lapply(X, FUN = globals::findGlobals)
```
We might be able to speed up `globals::findGlobals()` a bit here, ~~but don't know how much.~~ [UPDATE 2018-06-20]: there was a low-hanging fruit in the globals package making it possible to speed this up **lots**, cf. https://github.com/HenrikBengtsson/globals/commit/566e3e9395ab5050387396c894b06fb099f96a66. I'll be running revdep checks on globals (first and and second generation dependencies) to make sure this doesn't break anything. If all ok, the need for working around this in future.apply is much smaller.
~~Regardless, there could be a need for an argument controlling whether `X` should be searched for globals or not, especially since it is likely that in most use cases `X` does not have globals.~~
Contributor guide
Assessment
This issue has not been assessed yet.