NVIDIA / NVIDIA/cudf

Track admission backpressure per I/O task

Open
#23,812 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Right now there's no way to tell, per I/O task, whether it was admitted immediately or blocked waiting on `reserve_memory` (and for how long), versus some other cause of delay. Add that as a field alongside the `"IO Task"` log entry from #23811: something like a `backpressure_reason` (`memory_reserve`, `channel_full`, or similar) and a wait duration, recorded at the `reserve_memory` call sites (`streaming/actor_graph/io.py`, `streaming/actor_graph/join.py`).

This is the cudf-polars-side complement to rapidsai/rapidsmpf#1165. Without it, a task waiting on `memory_reserve` looks identical to a task waiting on the network, which is exactly the ambiguity that makes latency-only tuning unreliable.

Contributor guide

Open the contributing guide

Research direction

Start by reading the "IO Task" logging added in #23811, then inspect the reserve_memory call sites in streaming/actor_graph/io.py and streaming/actor_graph/join.py. Trace how each task records admission and waiting, and identify existing tests for these paths. Done means each I/O task log distinguishes memory-reserve backpressure from other delays and includes the corresponding wait duration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.