apache / apache/druid

[Druid Iceberg Extension] Implement incremental/snapshot-based ingestion to support real-time use cases

Open
#19,268 0 comments 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description
Currently, Druid performs full table scans on every ingestion run. For large Iceberg tables (billions of rows), this makes real-time ingestion impractical:
- Re-scanning 50TB table every minute is impossible
- Compute costs are prohibitive
- SLA requirements (sub-minute latency) cannot be met

### Motivation

A financial trading platform needs to ingest new stock trades within 30 seconds of arrival. Their Iceberg table has 50 billion historical rows.
Current Behavior:
ingestion:
type: iceberg
table: stock_trades
schedule: "@every 1m"

# Result:
# - Every 1 minute: Full table scan of 50 billion rows
# - Takes 45 minutes (FAILS SLA)
# - Cost: High

Contributor guide

Open the contributing guide

Research direction

Start by locating the Druid Iceberg extension's ingestion entry point and reviewing how each scheduled run currently performs a full table scan. Research Iceberg snapshot-based and incremental ingestion options, then define how new stock trades are detected without rescanning historical rows. Done means the ingestion design supports the stated sub-minute real-time use case while avoiding repeated scans of the full table.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.