futureverse / futureverse/future
Non-block batch assigns
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I have a project which requires loading of ~30GB data (~100 segments with each 300MB) into RAM, which is super time-consuming. The application only requires 5 segment of data to start analysis. I was wondering if future package can support non-blocked batch assignment. Basically lots of `future::futureAssign` in lapply way but non-blocking.
Right now if I use `multicore` plan, and set max worker number to be 8, then the 9th future will block the session. However, it could be best if this procedure is non-blocking. I implemented an ugly version using `future` and `later` packages, but you must have better idea of doing so.
The idea:
```r
future::plan(future::multiprocess)
a = new.env(parent = baseenv())
b = new.env(parent = baseenv())
a$plus = 10
# For each element of x, assign corresponding letter character to environment b with el + 10
NONBLOCKING_lapply(
x = 1:26, varnames = LETTERS[1:26], {
Sys.sleep(3)
return(el + plus)
}, nworkers = 2, envir = a, assign.env = b
)
Sys.sleep(3)
# Yay, I can start to view A in environment b at once
b$A
# >> 11
```
Contributor guide
Assessment
This issue has not been assessed yet.