apache / apache/datafusion-ballista

arrow-ipc-optimizations in ballista-executor does not forward to ballista-core

Open Beginner friendly
#2,368 0 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

Both `ballista-core` and `ballista-executor` declare a feature named `arrow-ipc-optimizations`, and both have it on by default:

- `ballista/core/Cargo.toml:38`, defaulted at line 40
- `ballista/executor/Cargo.toml:36`, defaulted at line 46

They are unrelated features that happen to share a name. The executor's gates exactly one call site, `flight_service.rs:159`. Core's gates the rest of the shuffle read path. The executor's does not forward to core's, unlike `spark-compat` right next to it at `ballista/executor/Cargo.toml:47`.

This makes `ballista/client/Cargo.toml:33-35` misleading. It pulls the executor with `default-features = false, features = ["arrow-ipc-optimizations"]`, which reads as "keep the IPC optimization on" but only enables the executor's half. Core's happens to be on anyway through the client's direct dependency on `ballista-core` at line 32, so nothing is broken today, but it works by accident.

Anyone building the executor standalone with `--no-default-features --features arrow-ipc-optimizations` gets one of the four decode sites optimized and the other three validating, with nothing to indicate it.

Suggested fix, in `ballista/executor/Cargo.toml`:

```toml
arrow-ipc-optimizations = ["ballista-core/arrow-ipc-optimizations"]
```

Default behavior is unchanged since both are already on by default. Worth noting in the fix that this still does not give a full off switch, because Cargo features are additive and anything else in the graph enabling core's default turns it back on.

Related: `examples/Cargo.toml:61` sets `default-features = false` on `ballista-core` to get `build-binary` and silently loses `arrow-ipc-optimizations` as a side effect. Same root cause, may be worth fixing in the same PR.

Contributor guide

Open the contributing guide

Research direction

Start with ballista/executor/Cargo.toml and compare its arrow-ipc-optimizations feature with the nearby spark-compat forwarding and ballista/core/Cargo.toml. Check the feature settings in ballista/client/Cargo.toml and examples/Cargo.toml, then verify Cargo builds with the relevant default and no-default feature combinations. Done means the executor feature forwards to core and the resulting behavior matches the stated default and standalone expectations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.