apache / apache/datafusion-ballista

Static job planner to respect `datafusion....hash_join_single_partition_threshold` for broadcast decision

Open
#1,770 5 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

static job planner uses `ballista.optimizer.broadcast_join_threshold_bytes` configuration value to decide if shuffle should be broadcast (`CollectLeft`), this configuration value diverges from datafusion configurations:

- `datafusion.optimizer.hash_join_single_partition_threshold_rows`
- `datafusion.optimizer.hash_join_single_partition_threshold`

which serve same purpose.

**Describe the solution you'd like**

I would propose to change static planner to to respect same configuration values like datafusion `datafusion.optimizer.hash_join_single_partition_threshold_rows` & `datafusion.optimizer.hash_join_single_partition_threshold`

AQE implementation (#1752) uses the same configuration values as datafusion hence it would be good if we can get both planners (and datafusion) aligned

suggestion is to change

`datafusion.optimizer.hash_join_single_partition_threshold_rows` https://github.com/apache/datafusion-ballista/blob/65ec1c479acf9fe7acb19a0780bc0a6100739b9c/ballista/core/src/extension.rs#L773 from 0 to 1M (datafusion `131072`)

and `datafusion.optimizer.hash_join_single_partition_threshold` https://github.com/apache/datafusion-ballista/blob/65ec1c479acf9fe7acb19a0780bc0a6100739b9c/ballista/core/src/extension.rs#L769 from 0 to 10MB (datafusion 1048576)

**Describe alternatives you've considered**

align both planners to `ballista.optimizer.broadcast_join_threshold_bytes` but that would diverge from datafusion configuration, which is not ideal as ballista tends to align with datafusion as much as possible

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start with the configuration definitions in ballista/core/src/extension.rs around lines 769-773, then trace how the static job planner makes its broadcast decision and compare it with the AQE implementation from #1752. Done means the static planner respects the DataFusion row and byte threshold settings and their defaults are aligned as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.