dask / dask/distributed

Include `BatchedSend` state in cluster dumps

Open
#6,114 0 comments 0 reactions 0 assignees View on GitHub
diagnostics
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

`BatchedSend` is known to be brittle, and certain error cases can cause it to drop messages or even shut down entirely, causing deadlocks:
* https://github.com/dask/distributed/issues/5481
* https://github.com/dask/distributed/pull/5457

I don't think we've seen these problems actually come up in the wild since https://github.com/dask/distributed/pull/5525.

But when looking over cluster dumps, it would be nice to be able to rule this out as an issue. So I think we should include some minimal information about the state of BatchedSend, such as:
* `len(buffer)`
* `please_stop`
* `waker.is_set()`
* `next_deadline`

Obviously it would also be nice to include the status of the coroutine, but since we don't have a handle to it, that's lost to the ether of Tornado.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.