apache / apache/datafusion-ballista
Surface collected column statistics to planners via ShuffleReaderExec
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
Follow-up to #1916 (part of #1827).
#1916 collects per-column null counts on the executor and stores them aggregated on `SuccessfulStage.output_column_stats` in the scheduler. They are not yet consumed: `ShuffleReaderExec::partition_statistics` (`stats_for_partition` in `shuffle_reader.rs`) still returns `column_statistics: vec![ColumnStatistics::new_unknown(); num_fields]`
1. `finalise_stage_internal` (`aqe/planner.rs`) — carry `output_column_stats` alongside partitions.
2. `resolve_shuffle_partitions` (`aqe/execution_plan/exchange.rs` + `adaptive.rs`) — store stats next to partition locations.
3. `BallistaAdapter::transform_children` (`aqe/adapter.rs`) — pass into `ShuffleReaderExec::try_new*`.
4. `ShuffleReaderExec` — new field plus proto/serde changes (serialized core plan node).
5. `stats_for_partition` (`shuffle_reader.rs`) — use the carried stats instead of `new_unknown()`.
6. Collect stats on the sort-based shuffle-write path (`sort_shuffle/writer.rs`, currently empty; TODO in code).
7. Collect stats on the single-partition write path (`shuffle_writer.rs`, currently empty; TODO in code).
Contributor guide
Assessment
This issue has not been assessed yet.