apache / apache/datafusion-ballista

Revisit the hard-coded default of 16 for datafusion.execution.target_partitions

Open
#2,128 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

## Is your feature request related to a problem or challenge?

`SessionConfig::new_with_ballista()` overrides `datafusion.execution.target_partitions` to a hard-coded `16`:

https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/extension.rs#L370-L377

DataFusion's own default is the number of CPU cores available to the process, which is not meaningful for Ballista because the work runs on executors rather than in the client process, so some override is needed. But a fixed `16` is arbitrary and, as noted in https://github.com/apache/datafusion-ballista/pull/2125#discussion_r3623918647, it may be too low for the clusters people actually run.

`target_partitions` is the main lever on scan parallelism in Ballista. File listing chunks a table's files into at most `target_partitions` file groups, and Ballista disables round robin repartitioning, so a scan will never produce more partitions than this setting no matter how much executor capacity is available. With the default, a 200 file table scanned on a cluster with 64 total vcores still produces only 16 partitions, and because `ballista.scheduler.max_partitions_per_task` defaults to `1`, that stage runs as 16 tasks and leaves most of the cluster idle. Users have to know to raise the setting before their cluster is used.

## Describe the solution you'd like

Revisit the default. A few directions worth considering:

1. Raise the constant to something larger that better matches typical cluster sizes.
2. Derive the default from cluster capacity (total executor vcores) at session creation time, rather than using a constant. This matches what the scheduler tests already do (`SessionConfig::new_with_ballista().with_target_partitions(total_vcores)`), but the scheduler would need to decide how to handle sessions created before executors register and clusters that scale up or down mid-session.
3. Keep a constant, but pick it deliberately and document the reasoning plus the interaction with `ballista.scheduler.max_partitions_per_task` in the tuning guide.

Whichever direction is chosen, the change should come with TPC-H numbers, since raising `target_partitions` increases shuffle partition counts as well as scan parallelism and that is not uniformly a win.

## Describe alternatives you've considered

Leaving the default at `16` and relying on documentation, which is the status quo after #2125.

## Additional context

Raised by @milenkovicm in review of #2125.

Contributor guide

Open the contributing guide

Research direction

Start in ballista/core/src/extension.rs around SessionConfig::new_with_ballista() and review the scheduler tests that set target_partitions from total_vcores. Compare the possible default strategies, including their interaction with ballista.scheduler.max_partitions_per_task, then run the relevant TPC-H benchmarks to determine whether the chosen approach improves scan parallelism without unacceptable shuffle costs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.