Producer-idle-time metric
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Scan/IO task assignment is fully static. `scan_node` (`streaming/actor_graph/io.py`) builds a fixed `producer_id = task_idx % num_producers` assignment once at graph-build time, no shared queue, no work-stealing. If one producer draws a large or slow file, other producers that finish early can't help. Log when a producer finishes its assigned list while others are still working. Quantifies the imbalance concretely and is useful evidence for (or against) a future queue-based scheduler change.
The actual shape of this metric depends on whatever queue-based scheduler redesign eventually replaces the static assignment, so hold off on locking in its exact form until that's picked.
On its own this metric can't say *why* a producer was slow. See #23814, a producer stuck waiting for a free thread there looks identical to one stuck on a genuinely large file.
Contributor guide
Research direction
Start with streaming/actor_graph/io.py, especially scan_node and its static producer assignment. Review the eventual queue-based scheduler redesign before defining the metric; done means logging when one producer finishes its assigned list while others are still working, without treating this metric as an explanation of the slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, observability, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100