apache / apache/datafusion-ballista
Revisit the hard-coded default of 16 for datafusion.execution.target_partitions
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
## Is your feature request related to a problem or challenge?
`SessionConfig::new_with_ballista()` overrides `datafusion.execution.target_partitions` to a hard-coded `16`:
https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/extension.rs#L370-L377
DataFusion's own default is the number of CPU cores available to the process, which is not meaningful for Ballista because the work runs on executors rather than in the client process, so some override is needed. But a fixed `16` is arbitrary and, as noted in https://github.com/apache/datafusion-ballista/pull/2125#discussion_r3623918647, it may be too low for the clusters people actually run.
`target_partitions` is the main lever on scan parallelism in Ballista. File listing chunks a table's files into at most `target_partitions` file groups, and Ballista disables round robin repartitioning, so a scan will never produce more partitions than this setting no matter how much executor capacity is available. With the default, a 200 file table scanned on a cluster with 64 total vcores still produces only 16 partitions, and because `ballista.scheduler.max_partitions_per_task` defaults to `1`, that stage runs as 16 tasks and leaves most of the cluster idle. Users have to know to raise the setting before their cluster is used.
## Describe the solution you'd like
Revisit the default. A few directions worth considering:
1. Raise the constant to something larger that better matches typical cluster sizes.
2. Derive the default from cluster capacity (total executor vcores) at session creation time, rather than using a constant. This matches what the scheduler tests already do (`SessionConfig::new_with_ballista().with_target_partitions(total_vcores)`), but the scheduler would need to decide how to handle sessions created before executors register and clusters that scale up or down mid-session.
3. Keep a constant, but pick it deliberately and document the reasoning plus the interaction with `ballista.scheduler.max_partitions_per_task` in the tuning guide.
Whichever direction is chosen, the change should come with TPC-H numbers, since raising `target_partitions` increases shuffle partition counts as well as scan parallelism and that is not uniformly a win.
## Describe alternatives you've considered
Leaving the default at `16` and relying on documentation, which is the status quo after #2125.
## Additional context
Raised by @milenkovicm in review of #2125.
Contributor guide
Research direction
Start in ballista/core/src/extension.rs around SessionConfig::new_with_ballista() and review the scheduler tests that set target_partitions from total_vcores. Compare the possible default strategies, including their interaction with ballista.scheduler.max_partitions_per_task, then run the relevant TPC-H benchmarks to determine whether the chosen approach improves scan parallelism without unacceptable shuffle costs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100