dask / dask/distributed

Forget provenance

Open
#809 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Iterative algorithms can have very long histories. Maintaining how we obtained final results in the scheduler can become very expensive.

```python
x = 1
for i in range(100000):
x = c.submit(inc, x)

wait(x)
```

In this case x only takes a few bytes in memory, but it's history in the scheduler takes several megabytes.

In cases like these it might be worthwhile to explicitly replace how we got `x` with a simple value, similar to what happens when we scatter data. This destroys options for resilience (if the worker goes down then we are unable to reproduce the value) but may still be useful.

```python
x = c.forget_provenance(x) # TODO: need better name
```

This would be helpful with iterative algorithms such as occur in `dask-glm` (cc @moody-marlin)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.