apache / apache/datafusion-ballista
Ballista: Fix hacks around concurrency=2 to force hash-partitioned joins
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
By default, DataFusion uses hash-partitioned joins if concurrency > 1 which led to me adding this hacky code in a couple of places in Ballista.
```rust
let config = ExecutionConfig::new().with_concurrency(2); // TODO: this is hack to enable partitioned joins
let mut ctx = ExecutionContext::with_config(config);
```
**Describe the solution you'd like**
I'm actually not sure what the solution should be, but I would like to be able to tell the context to use hash-partitioned joins, separately from specifying concurrency.
**Describe alternatives you've considered**
None
**Additional context**
This code is running in the scheduler, not in the executor where the query actually executes. The scheduler concurrency should not impact how the query is planned.
Contributor guide
Research direction
Start by tracing the scheduler's use of ExecutionConfig and ExecutionContext, then compare it with the executor path where the query is planned. Determine how hash-partitioned joins can be selected independently of scheduler concurrency; done means the scheduler no longer needs concurrency=2 as a planning hack.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100