apache / apache/datafusion-ballista
Static job planner to respect `datafusion....hash_join_single_partition_threshold` for broadcast decision
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
static job planner uses `ballista.optimizer.broadcast_join_threshold_bytes` configuration value to decide if shuffle should be broadcast (`CollectLeft`), this configuration value diverges from datafusion configurations:
- `datafusion.optimizer.hash_join_single_partition_threshold_rows`
- `datafusion.optimizer.hash_join_single_partition_threshold`
which serve same purpose.
**Describe the solution you'd like**
I would propose to change static planner to to respect same configuration values like datafusion `datafusion.optimizer.hash_join_single_partition_threshold_rows` & `datafusion.optimizer.hash_join_single_partition_threshold`
AQE implementation (#1752) uses the same configuration values as datafusion hence it would be good if we can get both planners (and datafusion) aligned
suggestion is to change
`datafusion.optimizer.hash_join_single_partition_threshold_rows` https://github.com/apache/datafusion-ballista/blob/65ec1c479acf9fe7acb19a0780bc0a6100739b9c/ballista/core/src/extension.rs#L773 from 0 to 1M (datafusion `131072`)
and `datafusion.optimizer.hash_join_single_partition_threshold` https://github.com/apache/datafusion-ballista/blob/65ec1c479acf9fe7acb19a0780bc0a6100739b9c/ballista/core/src/extension.rs#L769 from 0 to 10MB (datafusion 1048576)
**Describe alternatives you've considered**
align both planners to `ballista.optimizer.broadcast_join_threshold_bytes` but that would diverge from datafusion configuration, which is not ideal as ballista tends to align with datafusion as much as possible
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start with the configuration definitions in ballista/core/src/extension.rs around lines 769-773, then trace how the static job planner makes its broadcast decision and compare it with the AQE implementation from #1752. Done means the static planner respects the DataFusion row and byte threshold settings and their defaults are aligned as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100