[Druid Iceberg Extension] Implement incremental/snapshot-based ingestion to support real-time use cases
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
Currently, Druid performs full table scans on every ingestion run. For large Iceberg tables (billions of rows), this makes real-time ingestion impractical:
- Re-scanning 50TB table every minute is impossible
- Compute costs are prohibitive
- SLA requirements (sub-minute latency) cannot be met
### Motivation
A financial trading platform needs to ingest new stock trades within 30 seconds of arrival. Their Iceberg table has 50 billion historical rows.
Current Behavior:
ingestion:
type: iceberg
table: stock_trades
schedule: "@every 1m"
# Result:
# - Every 1 minute: Full table scan of 50 billion rows
# - Takes 45 minutes (FAILS SLA)
# - Cost: High
Contributor guide
Research direction
Start by locating the Druid Iceberg extension's ingestion entry point and reviewing how each scheduled run currently performs a full table scan. Research Iceberg snapshot-based and incremental ingestion options, then define how new stock trades are detected without rescanning historical rows. Done means the ingestion design supports the stated sub-minute real-time use case while avoiding repeated scans of the full table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100