apache / apache/datafusion-ballista
Python client silently forces the static planner (pinned to ballista 54.0.0 defaults)
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Describe the bug**
#2315 flipped `ballista.planner.adaptive.enabled` to default `true`, so a client built from `main` gets the adaptive (AQE) planner. The published Python client does not, and it silently forces the static planner instead.
`python/Cargo.toml` pins the Rust side to the released crates:
```toml
ballista = { version = "=54.0.0" }
ballista-core = { version = "=54.0.0" }
```
In 54.0.0 the default for that key is `false`. `BallistaConfig` materialises every setting including defaults and ships it to the scheduler with the session config, so the scheduler sees an explicit `ballista.planner.adaptive.enabled=false` and honours it. The user never asked for that, and there is nothing in the logs at INFO level to tell them which planner ran.
Confirmed against a cluster built at `apache/main` with scheduler debug logging on:
```
DEBUG ballista_scheduler::state::task_manager: Using adaptive query planner (AQE) for job planning <- Rust client from main
DEBUG ballista_scheduler::state::task_manager: Using static query planner for job planning <- ballista 54.0.0 from PyPI
```
Same query, same cluster, different planner purely because of which client submitted it.
**To Reproduce**
1. Build and start a scheduler and executor from `main`.
2. Run any TPC-H query with the Rust client, and again with `pip install ballista` (54.0.0 or 54.1.0).
3. With `RUST_LOG=ballista_scheduler::state::task_manager=debug` on the scheduler, compare the "query planner" lines.
**Expected behavior**
Python users should get the same planner as everyone else, and if a client does pin a non-default value it should be because the user asked for it.
Two things worth separating here:
1. Any config key whose default changes on the Rust side is invisible to a Python client pinned to an older release, and the mismatch is silent. It would be better if a client only sent keys the user actually set, so the scheduler's own defaults apply to everything else.
2. The version pin itself. `python/Cargo.toml` has the path dependencies commented out and depends on crates.io, so the Python bindings on `main` are not built against the code on `main`. That is fine for release builds but means the bindings lag every default and behaviour change until the next release.
Worth noting that this pin is not something we can simply drop. `pyballista` re-exports datafusion-python types (`datafusion_python::dataframe::PyDataFrame` and friends in `python/src/lib.rs`), so `datafusion-python`, `datafusion`, and `ballista-core` all have to link the same `datafusion`. The workspace is currently on the `55.0.0-rc3` git tag while crates.io `datafusion-python` is still at `54.0.0`, so the bindings cannot move until there is a matching datafusion-python release. In practice the Python client will normally lag the cluster by a DataFusion release cycle, which makes fix (1) above the one that actually helps.
**Additional context**
Found while investigating #2367. Related: #2315, and the CI gap in #2372.
Contributor guide
Research direction
Start with python/Cargo.toml and python/src/lib.rs, then trace how BallistaConfig materialises settings before they reach the scheduler. Reproduce the planner mismatch using the TPC-H query steps and task_manager debug logs. Done means an unset planner setting no longer overrides the scheduler default, with the client and scheduler selecting the same planner unless the user explicitly configures one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100