apache / apache/druid

Iceberg extension quietly ignores delete markers resulting in incorrect data

Open
#18,858 0 comments 0 reactions 0 assignees View on GitHub
Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

If an iceberg table using merge-on-read updates or deletes is ingested into druid, then the deleted rows will be ingested as well.

As a simple example, we can create a quick Iceberg table using Spark:

```scala
val df = Seq(
("store_a", 1, 100),
("store_a", 2, 200),
("store_b", 3, 300),
("store_b", 4, 400),
).toDF("store_id", "item_count", "price_total")

df.withColumn("ts", current_timestamp()).
writeTo("demo.test_database.checkouts").
using("iceberg").
partitionedBy(hours($"ts")).
tableProperty("write.update.mode", "merge-on-read").
create()
```

Then update the table:
```sql
UPDATE demo.test_database.checkouts SET total_price=0 WHERE store_id = 'store_a'
```

Ingesting the table into druid then shows 6 rows, due to ingesting both versions of the updated records:
```sql
SELECT * FROM "checkouts"

{"__time":"2025-12-19T00:00:00.000Z","store_id":"store_a","count":4,"sum_item_count":6,"sum_price_total":300}
{"__time":"2025-12-19T00:00:00.000Z","store_id":"store_b","count":2,"sum_item_count":7,"sum_price_total":700}
```

This feels like a potential hazard which isn't explicitly called out in [the documentation](https://druid.apache.org/docs/latest/development/extensions-contrib/iceberg/).

Ideally we would handle the delete markers and properly materialize the data, but thats a pretty big overhaul. As a more simple solution should we maybe just fail the ingestion if there's delete markers present in the target partitions?

Happy to help with the implementation here, or at least just updating the documentation to make this more clear - let me know what you think is the best path forwards.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Iceberg extension documentation and reproducing the Spark merge-on-read example described here. Clarify whether the scope is materializing delete markers, failing ingestion, or documenting the limitation; completion should be defined before implementation begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.