[Story]: Instrument I/O concurrency and partition-sizing signals across cudf-polars, rapidsmpf, kvikio
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Problem
Tuning cudf-polars' I/O and memory-admission settings today is mostly guesswork. `max_concurrent_io_tasks` (#23552) and `target_partition_size` (#23549) are what prompted this, but the same blind spot applies to `num_py_executors`, `kvikio_nthreads`, pinned memory, and any future auto-tuning work. We audited cudf-polars, rapidsmpf, and kvikio and found the same pattern in all three: the raw signal usually already exists somewhere in the C++ layer, but is discarded, folded into an aggregate that's too coarse, or never wired up. Most of this list is "turn on a light that's already wired," not new sensor design.
Goal: give sweeps, benchmark runs, and future tuning work a way to tell what's actually causing a result, instead of guessing.
## Related
- #23552 `max_concurrent_io_tasks` default
- #23549 `target_partition_size` default
- rapidsai/kvikio#999, `kvikio_nic`: an nsys plugin that samples per-interface NIC bandwidth and places it on the nsys timeline alongside CUDA/NVTX traces.
- rapidsai/kvikio#1033, `Add an I/O observation facility`, and rapidsai/kvikio#1036, `Add I/O statistics: totals of what a run read and wrote`: the `Observation`/`Summary`/`SummaryMonitor` machinery the kvikio sub-issues in this story build on.
- #23179 and #22949: an active Quent tracing effort adding resource-level instrumentation (CPU, device memory, filesystem memory, network) to cudf-polars. The signals this story adds may be useful to surface in Quent.
- #23738, `Report kvikio I/O statistics per rank in the streaming engines`: recent cudf-polars work surfacing kvikio's `Summary` per rank, the current baseline the kvikio sub-issues in this story go beyond.
- rapidsai/rapidsmpf#875, `Introduce Statistics::record_copy and consolidate copy/spill statistics`, and rapidsai/rapidsmpf#882, `Stream ordered timings`: the most recent rapidsmpf statistics work, and the direct predecessor of rapidsai/rapidsmpf#1166.
- rapidsai/rapidsmpf#869 (open), `cudf-polars 1k ndsh runs show malformed spill stats`: a live report of the same gap rapidsai/rapidsmpf#1166 is about.
- rapidsai/rapidsmpf#1155 (open), `[Discussion] Implement backpressure when allocating buffers for incoming payloads on a rank`: related to rapidsai/rapidsmpf#1165, though scoped to the shuffler specifically rather than I/O reads generally.
Contributor guide
Research direction
Start by reading the related issues #23552 and #23549, then review the referenced kvikio Observation/Summary work and rapidsmpf statistics issues. The story spans cudf-polars, rapidsmpf, and kvikio but names no files or tests, so identify a concrete sub-issue and entry point before coding. Done means the required I/O and partition-sizing signals are exposed for tuning and benchmark runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100