Log the actual output size of each I/O task
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
`read_chunk` (`streaming/actor_graph/io.py`) already logs `start`/`admitted`/`stop` timestamps, `estimated_output_bytes`, and `reservation_bytes` per I/O task. Missing one field: `actual_output_bytes`. Needed to calibrate `target_partition_size` against real outcomes and to check the fixed `2x` memory-reservation multiplier against real usage. The value is already computed once the chunk is materialized, this is just plumbing it into the existing log call.
Longer term, #20780 (`cudf::memory_resources`, separating output allocations from temporary/intermediate ones in libcudf APIs) would let us measure the temporary allocation a parquet read actually uses instead of assuming the fixed `2x`, once the IO module is migrated to it.
Contributor guide
Research direction
Open streaming/actor_graph/io.py and start at read_chunk, following where the chunk is materialized and where the existing per-task log record is assembled. Add the already-computed output size to that log record, then verify that each I/O task emits actual_output_bytes alongside the existing fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 90/100