apache / apache/datafusion-comet

Broaden storage-partitioned-join tests, then enable reportPartitioning by default

Open
#5,342 0 comments 0 reactions 1 assignee Claimed by @parthchandra View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow up for #5323

The grouping report (reportPartitioning.enabled) ships off by default until we've covered more of Spark's storage-partitioned-join paths. Add tests for:
- a heavily skewed key, so the replicate path actually runs (not just empty-partition padding);
- different bucket counts per side (e.g. bucket(8) vs bucket(4)), so Spark inserts a partition reducer and the Comet scan must match the reduced grouping;
- three-way and self joins on the same key.

Once these tests pass, then we can flip `reportPartitioning.enabled` on by default

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.