Track admission backpressure per I/O task
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Right now there's no way to tell, per I/O task, whether it was admitted immediately or blocked waiting on `reserve_memory` (and for how long), versus some other cause of delay. Add that as a field alongside the `"IO Task"` log entry from #23811: something like a `backpressure_reason` (`memory_reserve`, `channel_full`, or similar) and a wait duration, recorded at the `reserve_memory` call sites (`streaming/actor_graph/io.py`, `streaming/actor_graph/join.py`).
This is the cudf-polars-side complement to rapidsai/rapidsmpf#1165. Without it, a task waiting on `memory_reserve` looks identical to a task waiting on the network, which is exactly the ambiguity that makes latency-only tuning unreliable.
Contributor guide
Research direction
Start by reading the "IO Task" logging added in #23811, then inspect the reserve_memory call sites in streaming/actor_graph/io.py and streaming/actor_graph/join.py. Trace how each task records admission and waiting, and identify existing tests for these paths. Done means each I/O task log distinguishes memory-reserve backpressure from other delays and includes the corresponding wait duration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100