apache / apache/druid

[Druid Iceberg Extension] Add column projection support to reduce I/O and improve query performance

Open
#19,267 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Currently, the Druid Iceberg extension reads ALL columns from Iceberg data files regardless of which columns are needed for ingestion. For tables with hundreds of columns, this causes:
- 10-100x unnecessary data read from storage
- Increased memory pressure during ingestion
- Slower query performance
- Higher cloud storage egress costs

An e-commerce analytics team has an Iceberg table with 150 columns but only needs 5 columns (timestamp, product_id, category, price, quantity) for their Druid dashboard. Currently, Druid reads all 150 columns, causing:
- Query time:
- Memory:
- Data transfer:

Contributor guide

Open the contributing guide

Research direction

Start by locating the Druid Iceberg extension and its ingestion path, then trace how the columns required by ingestion are selected and read. Define the projection boundary from the issue's requested fields; done should mean unused Iceberg columns are not read, with coverage for a table where only a subset of columns is needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.