Forget provenance
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Iterative algorithms can have very long histories. Maintaining how we obtained final results in the scheduler can become very expensive.
```python
x = 1
for i in range(100000):
x = c.submit(inc, x)
wait(x)
```
In this case x only takes a few bytes in memory, but it's history in the scheduler takes several megabytes.
In cases like these it might be worthwhile to explicitly replace how we got `x` with a simple value, similar to what happens when we scatter data. This destroys options for resilience (if the worker goes down then we are unable to reproduce the value) but may still be useful.
```python
x = c.forget_provenance(x) # TODO: need better name
```
This would be helpful with iterative algorithms such as occur in `dask-glm` (cc @moody-marlin)
Contributor guide
Assessment
This issue has not been assessed yet.