Worker crash causes computations to overlap
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
1. Persist and wait a computation
2. Start a second computation
3. Kill a worker that holds some of the futures of the first computation, causing them to be recomputed
This causes the `stop` of the first computation to be after the `stop` of the second computation.
In other words, a computation other than `Scheduler.computations[-1]` was the last one to accrue work.
```python
import os
import distributed
client = distributed.Client(n_workers=2)
a, b = (n.worker_address for n in client.cluster.workers.values())
x = client.submit(lambda: 1, key="x")
x.result()
has_x = client.who_has()["x"][0]
has_y = b if has_x == a else a
y = client.submit(lambda: 2, key="y", workers=[has_y])
y.result()
# Force x to be recomputed
def seppuku():
os.kill(os.getpid(), 9)
try:
client.run(seppuku, workers=[has_x])
except Exception:
pass
x.result()
[(c.start, c.stop) for c in client.cluster.scheduler.computations]
[(1683718227.5482748, 1683718230.0930064),
(1683718227.5749886, 1683718227.684088)]
```
This is problematic when you want to apportion data to a computation without knowing the task groups, e.g. #7776.
XREFs
- #7776
- #7787
- #7790
Contributor guide
Assessment
This issue has not been assessed yet.