futureverse / futureverse/future

Should memoization be added?

Open
#127 1 comment 4 reactions 0 assignees View on GitHub
question
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Moving an old Issue https://github.com/HenrikBengtsson/future.BatchJobs/issues/20 to here where it belong. I'm just leaving this here for a discussion for now; no immediate plans for implementing it.

Does it make sense to add support for memoization to futures? For instance,
```r
f <- future({ ... }, memoize = TRUE)
```
and
```r
x %<-% { ... } %memoize% TRUE
```

If so, then the memoization should ideally account for:

1. the future expression
2. all global variables
3. all package dependencies (and their versions)

Give how futures and globals work now, the above shouldn't be too hard.

One also needs to consider RNG states, i.e. does the value of the future depend on the current RNG state? In other words, one needs to be able to specify that too somehow, e.g. `memoize = list(rng = TRUE)` and `memoize = list(rng = FALSE)`.

Also, the memoization cache can be stored in RAM or in a persistent storage such as to file, cf. [R.cache](https://cran.r-project.org/package=R.cache). If storing to to persistent storage, then other things such as expiration dates and file cleanups have to be considered.

What else needs to be considered?

Contributor guide

Open the contributing guide

Research direction

No implementation file, test, or entry point is identified. Treat this as a design discussion: review the future() and %<- syntax mentioned, then define how expressions, globals, package versions, RNG state, and RAM or persistent-cache behavior would be specified and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.