apache / apache/datafusion

`RepartitionExec` is missing `BaselineMetrics`

Open
#18,218 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`BaselineMetrics` is a set of common metrics for almost all operators, now `RepartitionExec` is not tracking it.

The `BaselineMetrics` `output_rows` and `elapsed_compute` are not showed in `RepartitionExec` in the below `datafusion-cli` reproducer:
```
> explain analyze
select v1, count(*)
from generate_series(1,100) as t1(v1)
group by v1;
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[v1@0 as v1, count(Int64(1))@1 as count(*)], metrics=[output_rows=100, elapsed_compute=9.332µs] |
| | AggregateExec: mode=FinalPartitioned, gby=[v1@0 as v1], aggr=[count(Int64(1))], metrics=[output_rows=100, elapsed_compute=4.043955ms, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=67328] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=100, elapsed_compute=60.834µs] |
| | RepartitionExec: partitioning=Hash([v1@0], 14), input_partitions=14, metrics=[fetch_time=7.515374ms, repartition_time=45.428µs, send_time=14.348µs] |
| | AggregateExec: mode=Partial, gby=[v1@0 as v1], aggr=[count(Int64(1))], metrics=[output_rows=100, elapsed_compute=5.612461ms, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0, peak_mem_used=7328] |
| | RepartitionExec: partitioning=RoundRobinBatch(14), input_partitions=1, metrics=[fetch_time=17.459µs, repartition_time=1ns, send_time=11.471µs] |
| | ProjectionExec: expr=[value@0 as v1], metrics=[output_rows=100, elapsed_compute=1.667µs] |
| | LazyMemoryExec: partitions=1, batch_generators=[generate_series: start=1, end=100, batch_size=8192], metrics=[output_rows=100, elapsed_compute=10.458µs] |
| | |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.
```

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the RepartitionExec implementation and how BaselineMetrics is used by the other operators. Reproduce the issue with the provided datafusion-cli EXPLAIN ANALYZE query, then verify that RepartitionExec reports output_rows and elapsed_compute alongside its existing metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.