futureverse / futureverse/future

Non-block batch assigns

Open
#307 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hi, I have a project which requires loading of ~30GB data (~100 segments with each 300MB) into RAM, which is super time-consuming. The application only requires 5 segment of data to start analysis. I was wondering if future package can support non-blocked batch assignment. Basically lots of `future::futureAssign` in lapply way but non-blocking.

Right now if I use `multicore` plan, and set max worker number to be 8, then the 9th future will block the session. However, it could be best if this procedure is non-blocking. I implemented an ugly version using `future` and `later` packages, but you must have better idea of doing so.

The idea:
```r
future::plan(future::multiprocess)

a = new.env(parent = baseenv())
b = new.env(parent = baseenv())
a$plus = 10

# For each element of x, assign corresponding letter character to environment b with el + 10
NONBLOCKING_lapply(
x = 1:26, varnames = LETTERS[1:26], {
Sys.sleep(3)
return(el + plus)
}, nworkers = 2, envir = a, assign.env = b
)
Sys.sleep(3)

# Yay, I can start to view A in environment b at once
b$A
# >> 11
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.