apache / apache/datafusion-ballista

Ballista: Fix hacks around concurrency=2 to force hash-partitioned joins

Open
#20 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

By default, DataFusion uses hash-partitioned joins if concurrency > 1 which led to me adding this hacky code in a couple of places in Ballista.

```rust
let config = ExecutionConfig::new().with_concurrency(2); // TODO: this is hack to enable partitioned joins
let mut ctx = ExecutionContext::with_config(config);
```

**Describe the solution you'd like**
I'm actually not sure what the solution should be, but I would like to be able to tell the context to use hash-partitioned joins, separately from specifying concurrency.

**Describe alternatives you've considered**
None

**Additional context**
This code is running in the scheduler, not in the executor where the query actually executes. The scheduler concurrency should not impact how the query is planned.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the scheduler's use of ExecutionConfig and ExecutionContext, then compare it with the executor path where the query is planned. Determine how hash-partitioned joins can be selected independently of scheduler concurrency; done means the scheduler no longer needs concurrency=2 as a planning hack.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.