NVIDIA / NVIDIA/cudf

[Story]: Instrument I/O concurrency and partition-sizing signals across cudf-polars, rapidsmpf, kvikio

Open
#23,810 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Problem

Tuning cudf-polars' I/O and memory-admission settings today is mostly guesswork. `max_concurrent_io_tasks` (#23552) and `target_partition_size` (#23549) are what prompted this, but the same blind spot applies to `num_py_executors`, `kvikio_nthreads`, pinned memory, and any future auto-tuning work. We audited cudf-polars, rapidsmpf, and kvikio and found the same pattern in all three: the raw signal usually already exists somewhere in the C++ layer, but is discarded, folded into an aggregate that's too coarse, or never wired up. Most of this list is "turn on a light that's already wired," not new sensor design.

Goal: give sweeps, benchmark runs, and future tuning work a way to tell what's actually causing a result, instead of guessing.

## Related

- #23552 `max_concurrent_io_tasks` default
- #23549 `target_partition_size` default
- rapidsai/kvikio#999, `kvikio_nic`: an nsys plugin that samples per-interface NIC bandwidth and places it on the nsys timeline alongside CUDA/NVTX traces.
- rapidsai/kvikio#1033, `Add an I/O observation facility`, and rapidsai/kvikio#1036, `Add I/O statistics: totals of what a run read and wrote`: the `Observation`/`Summary`/`SummaryMonitor` machinery the kvikio sub-issues in this story build on.
- #23179 and #22949: an active Quent tracing effort adding resource-level instrumentation (CPU, device memory, filesystem memory, network) to cudf-polars. The signals this story adds may be useful to surface in Quent.
- #23738, `Report kvikio I/O statistics per rank in the streaming engines`: recent cudf-polars work surfacing kvikio's `Summary` per rank, the current baseline the kvikio sub-issues in this story go beyond.
- rapidsai/rapidsmpf#875, `Introduce Statistics::record_copy and consolidate copy/spill statistics`, and rapidsai/rapidsmpf#882, `Stream ordered timings`: the most recent rapidsmpf statistics work, and the direct predecessor of rapidsai/rapidsmpf#1166.
- rapidsai/rapidsmpf#869 (open), `cudf-polars 1k ndsh runs show malformed spill stats`: a live report of the same gap rapidsai/rapidsmpf#1166 is about.
- rapidsai/rapidsmpf#1155 (open), `[Discussion] Implement backpressure when allocating buffers for incoming payloads on a rank`: related to rapidsai/rapidsmpf#1165, though scoped to the shuffler specifically rather than I/O reads generally.

Contributor guide

Open the contributing guide

Research direction

Start by reading the related issues #23552 and #23549, then review the referenced kvikio Observation/Summary work and rapidsmpf statistics issues. The story spans cudf-polars, rapidsmpf, and kvikio but names no files or tests, so identify a concrete sub-issue and entry point before coding. Done means the required I/O and partition-sizing signals are exposed for tuning and benchmark runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.