apache / apache/iceberg

Unexpected duplicate ID after ingest: input parquet has single ID, but Iceberg table keeps one duplicated key

Open
#16,685 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

None

### Query engine

None

### Please describe the bug 🐞

Hi Iceberg community,

I’m investigating a persistent duplicate-key issue and wanted to check if this matches any known Iceberg behavior.

Context:
- We ingest parquet data into an Iceberg table using data-load and incremental-load jobs.
- Source parquet for the affected ID has a single record (verified).
- After ingest, the Iceberg table shows exactly one duplicated ID key (same specific ID).
- We deleted that ID from the branch/table and ingested again, but the same duplicate reappeared.

Observed behavior:
- This is not a broad duplicate problem; it is isolated to one ID key.
- Snapshot row counts change across commits, and only this key is duplicated intermittently.
- Ingest is expected to be idempotent for this key pattern, but it is not.

Runtime/version detail:
- Spark: 3.5.x
- Iceberg :1.9.2

Questions:

Are there known cases where MERGE/UPSERT into Iceberg can produce a persistent single-key duplicate like this?
What metadata checks should we run/share first to determine if this is Iceberg-level vs ingest-logic-level?

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the ingest with Spark 3.5.x and Iceberg 1.9.2, then inspect the affected snapshots and table metadata around the commits where the key duplicates. Compare the MERGE/UPSERT behavior with the data-load and incremental-load jobs. Done means identifying whether the duplicate originates in Iceberg or in the ingest logic and documenting the relevant evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.