Add Granular Metrics to RepartitionExec
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`RepartitionExec` can be expensive in some workloads, but the existing metrics make it hard to tell where time is spent. Broad timings are useful for normal `EXPLAIN ANALYZE`, but investigating repartition bottlenecks needs lower-level timings without adding noise or overhead to the default output.
### Describe the solution you would like
Add granular `RepartitionExec` timing metrics that separate the major phases of the operator. The intended metrics are:
Default `dev` metrics:
- `fetch_time`: time spent polling input partitions for batches.
- `repartition_time`: end-to-end time spent repartitioning input batches, excluding input fetch and output channel send time.
- `send_time`: time spent preparing and sending partitioned batches to each output partition channel.
`internal` metrics:
- `hash_compute_time`: hash repartition only; time spent evaluating partitioning expressions and computing row hashes.
- `route_time`: hash repartition only; time spent assigning hashed row indices to output partitions.
- `batch_build_time`: hash repartition only; time spent materializing output batches from routed row indices.
- `channel_wait_time`: time spent waiting for output channel capacity while sending partitioned batches.
- `spill_write_time`: time spent writing partitioned batches to spill storage when memory reservation for an output channel is full.
### Describe alternatives you have considered
Adding more counters for partitioned or sent batches and rows was considered, but those values duplicate information available elsewhere and add cardinality without enough diagnostic value. Keeping the detailed metrics as timing-only internals should make the output more useful and easier to interpret.
### Additional context
These metrics are intended to pair with an `internal` `EXPLAIN ANALYZE` level so low-level debugging metrics are opt-in.
Contributor guide
Research direction
Start with the RepartitionExec implementation and its existing metrics, then trace how metrics are exposed for EXPLAIN ANALYZE. Define the requested default dev timings separately from internal timings, and verify that detailed metrics appear only for internal output while normal output remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100