Improvements to knowing why a task is blocked
- Dominant language
- Rust
- Stars
- 1.1k
- Forks
- 59
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 15
Description
Currently when a task gets blocked, we have one of the following situations:
1. The task is blocked due to using a primitive from `shuttle::sync`.
2. The task is blocked due to being blocked on a `Future` which is `Poll::Pending`.
For situation 1 we know for sure why the task is blocked, while for situation 2 we do not, as it bubbles up as the task being blocked on an opaque `Future` in `shuttle::future::block_on`.
Some comments:
1. Many of the async primitives have sync ways to acquire the primitive (eg. `tokio::sync::Mutex::blocking_lock`) that currently are implemented by doing a `shuttle::future::block_on` on the async variant (eg. `tokio::sync::Mutex::blocking_lock`). If we had only used sync methods then we would have more information about why the task is blocked.
2. There should be a better "reverse mapping" from `Acquire`s to tasks, so that we can log which permits a task holds (and also what primitives those permits are related to) and which permits its wanting to acquire (and this information should be displayed on a deadlock).
3. If we expose "building blocks" for creating Shuttle primitives then we should take some care in making it easy to have good debuggability.
4. I don't see a great way to fully "solve" `block_on`; we should for sure `track_caller`, but that still only tells us what we are blocking on. Let's say we are blocking on a `FuturesUnordered` then ideally we'd be able to investigate the state of all `Future`s in it.
5. We can get slightly more information for `Future`s by adding callbacks when a `Waker` is `clone`d (https://github.com/awslabs/shuttle/blob/49faea29d58a05f9e5c5b3bde89f6cd5636ee3bd/shuttle/src/runtime/task/waker.rs#L55), under the assumption that that is done right before a `Future` becomes `Poll::Pending`
Contributor guide
Research direction
Start with shuttle::future::block_on and the task waker implementation in shuttle/src/runtime/task/waker.rs, then review how Acquire instances map to tasks. The issue describes several possible improvements rather than a defined change; done would require choosing and implementing a concrete way to expose why a task is blocked, including useful deadlock information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100