apache / apache/auron

Native Iceberg scan returns incorrect results for storage-partitioned joins

Open
#2,390 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Describe the bug**

Native Iceberg scans do not preserve Spark's key-grouped input partitions. This can cause incorrect results in storage-partitioned joins.

**To Reproduce**

Create two Iceberg tables partitioned by the join key:

```sql
CREATE TABLE local.db.t_left (id INT, p INT)
USING iceberg
PARTITIONED BY (p);

INSERT INTO local.db.t_left VALUES (0, 0), (1, 1);

CREATE TABLE local.db.t_right (value INT, p INT)
USING iceberg
PARTITIONED BY (p);

INSERT INTO local.db.t_right VALUES (10, 0), (11, 0), (12, 1);
```

Enable storage-partitioned joins:

spark.sql.sources.v2.bucketing.enabled=true
spark.sql.iceberg.planning.preserve-data-grouping=true

Run a sort-merge join:

```sql
SELECT /*+ MERGE(l, r) */ l.id, l.p, r.value
FROM local.db.t_left l
JOIN local.db.t_right r ON l.p = r.p;
```

**Expected behavior**

The query returns:

(0, 0, 10)
(0, 0, 11)
(1, 1, 12)

**Actual result**

The query returns only one row:

(0, 0, 11)

The expected rows (0, 0, 10) and (1, 1, 12) are missing.

**Screenshots**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by running the SQL reproduction with storage-partitioned joins and the two Iceberg tables described in the issue. Trace how native Iceberg scan planning preserves Spark's key-grouped input partitions, then verify the sort-merge join returns all three expected rows instead of only one.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark, sql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.