NVIDIA / NVIDIA/cudf

Log the actual output size of each I/O task

Open Beginner friendly
#23,811 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

`read_chunk` (`streaming/actor_graph/io.py`) already logs `start`/`admitted`/`stop` timestamps, `estimated_output_bytes`, and `reservation_bytes` per I/O task. Missing one field: `actual_output_bytes`. Needed to calibrate `target_partition_size` against real outcomes and to check the fixed `2x` memory-reservation multiplier against real usage. The value is already computed once the chunk is materialized, this is just plumbing it into the existing log call.

Longer term, #20780 (`cudf::memory_resources`, separating output allocations from temporary/intermediate ones in libcudf APIs) would let us measure the temporary allocation a parquet read actually uses instead of assuming the fixed `2x`, once the IO module is migrated to it.

Contributor guide

Open the contributing guide

Research direction

Open streaming/actor_graph/io.py and start at read_chunk, following where the chunk is materialized and where the existing per-task log record is assembled. Add the already-computed output size to that log record, then verify that each I/O task emits actual_output_bytes alongside the existing fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
90/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.