dask / dask/distributed

Re-submitting tasks can creates `Computation`s with no `stop`

Open
#7,662 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

`Computation`s are meant to track, high-level, the operations that ran for a given `compute`/`persist` call. They're currently just a container referencing multiple `TaskGroup`s. But the same `TaskGroup` could be used in multiple, separate `compute` calls. For example:

```python
a = da.ones(1000)
h = a[:50].persist()
wait(h)
a = a.persist()
```

In this case, there will be two `Computation`s for the two `persist` calls. However, the second `persist` call re-uses the same `TaskGroup`s as the first. So its `groups` will be empty. Since the `stop` time is defined as when the last group stopped, the second computation will have no stop.

A test:

```python
@gen_cluster(client=True)
async def test_computations_some_resubmitted_tasks(c, s, a, b):
futures = c.map(inc, list(range(4)))
await wait(futures)

futures2 = c.map(inc, list(range(6))) # NOTE: 4 tasks reused, 2 new
await wait(futures2)

assert len(s.computations) == 2
assert s.computations[0].stop > s.computations[0].start
assert s.computations[1].stop > s.computations[1].start
assert s.computations[1].start > s.computations[0].stop
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.