apache / apache/datafusion-ballista

Surface collected column statistics to planners via ShuffleReaderExec

Open
#1,960 0 comments 0 reactions 1 assignee Claimed by @coderfender View on GitHub
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

Follow-up to #1916 (part of #1827).

#1916 collects per-column null counts on the executor and stores them aggregated on `SuccessfulStage.output_column_stats` in the scheduler. They are not yet consumed: `ShuffleReaderExec::partition_statistics` (`stats_for_partition` in `shuffle_reader.rs`) still returns `column_statistics: vec![ColumnStatistics::new_unknown(); num_fields]`

1. `finalise_stage_internal` (`aqe/planner.rs`) — carry `output_column_stats` alongside partitions.
2. `resolve_shuffle_partitions` (`aqe/execution_plan/exchange.rs` + `adaptive.rs`) — store stats next to partition locations.
3. `BallistaAdapter::transform_children` (`aqe/adapter.rs`) — pass into `ShuffleReaderExec::try_new*`.
4. `ShuffleReaderExec` — new field plus proto/serde changes (serialized core plan node).
5. `stats_for_partition` (`shuffle_reader.rs`) — use the carried stats instead of `new_unknown()`.
6. Collect stats on the sort-based shuffle-write path (`sort_shuffle/writer.rs`, currently empty; TODO in code).
7. Collect stats on the single-partition write path (`shuffle_writer.rs`, currently empty; TODO in code).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.