Unexpected duplicate ID after ingest: input parquet has single ID, but Iceberg table keeps one duplicated key
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
None
### Query engine
None
### Please describe the bug 🐞
Hi Iceberg community,
I’m investigating a persistent duplicate-key issue and wanted to check if this matches any known Iceberg behavior.
Context:
- We ingest parquet data into an Iceberg table using data-load and incremental-load jobs.
- Source parquet for the affected ID has a single record (verified).
- After ingest, the Iceberg table shows exactly one duplicated ID key (same specific ID).
- We deleted that ID from the branch/table and ingested again, but the same duplicate reappeared.
Observed behavior:
- This is not a broad duplicate problem; it is isolated to one ID key.
- Snapshot row counts change across commits, and only this key is duplicated intermittently.
- Ingest is expected to be idempotent for this key pattern, but it is not.
Runtime/version detail:
- Spark: 3.5.x
- Iceberg :1.9.2
Questions:
Are there known cases where MERGE/UPSERT into Iceberg can produce a persistent single-key duplicate like this?
What metadata checks should we run/share first to determine if this is Iceberg-level vs ingest-logic-level?
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by reproducing the ingest with Spark 3.5.x and Iceberg 1.9.2, then inspect the affected snapshots and table metadata around the commits where the key duplicates. Compare the MERGE/UPSERT behavior with the data-load and incremental-load jobs. Done means identifying whether the duplicate originates in Iceberg or in the ingest logic and documenting the relevant evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100