apache / apache/datafusion

manual repartitioning overridden by physical optimizer

Open
#3,629 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Describe the bug**

Using dataframe.repartition() function doesn't work as expected.

**To Reproduce**
Using the tpch bin from benchmarks, convert the .tbl (csv) files to Parquet format using the "partitions" option:

```
cargo run --release --bin tpch -- convert --input ./tpch-data --output ./tpch-data-parquet --format parquet --partitions 4
```

That should have produced 4 parquet files per table, but instead created 20 (this laptop has 20 cores).

**Expected behavior**

Expected it to produce the specified number of partitions (4 in this case).

**Additional context**
I added some debug and the physical plan produced is:
```
CoalesceBatchesExec: target_batch_size=4096
RepartitionExec: partitioning=RoundRobinBatch(20)
RepartitionExec: partitioning=RoundRobinBatch(4)
RepartitionExec: partitioning=RoundRobinBatch(20)
CsvExec: files=[home/kmitchener/dev/arrow-datafusion/benchmarks/tpch-data/region.tbl], has_header=false, limit=None, projection=[r_regionkey, r_name, r_comment]
```

The tpch bin repartitions the file using this bit of code:
```rust
// optionally, repartition the file
if opt.partitions > 1 {
csv = csv.repartition(Partitioning::RoundRobinBatch(opt.partitions))?
}
```

Contributor guide

Open the contributing guide

Research direction

Start by running the benchmarks/tpch binary with the reproduction command and inspect the physical plan around dataframe.repartition(). Trace why the physical optimizer adds RoundRobinBatch(20) repartitions after the requested 4-partition repartition. Done means the command produces four Parquet files per table as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.