apache / apache/druid

Add ORC and Avro file format support to Druid's Iceberg input source

Open
#19,472 1 comment 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

Component: extensions-contrib/druid-iceberg-extensions

Druid's Iceberg input source (druid-iceberg-extensions) currently only supports reading Iceberg tables stored in Parquet format.

IcebergNativeRecordReader hardcodes Parquet.read() + GenericParquetReaders for all reads:

### Motivation

This was flaged while working on v2 spec support https://github.com/apache/druid/pull/19266#discussion_r3259155168

dependecies : iceberg-orc, orc-core, iceberg-avro, and avro are absent

### References:

• IcebergNativeRecordReader.java — current Parquet-only implementation
• IcebergFileTaskInputSource.java — serialisation boundary between coordinator and worker
• Iceberg API: org.apache.iceberg.data.GenericDeleteFilter (public, already on classpath)
• Iceberg API: org.apache.iceberg.data.orc.GenericOrcReader, org.apache.iceberg.data.avro.GenericAvroReader
• PR #19266 — added Iceberg V2 delete support (Parquet only); this is the follow-up

Contributor guide

Open the contributing guide

Research direction

Start with extensions-contrib/druid-iceberg-extensions/IcebergNativeRecordReader.java to trace the Parquet-only read path, then inspect IcebergFileTaskInputSource.java for the coordinator-to-worker serialization boundary. Review the referenced GenericOrcReader and GenericAvroReader APIs and the missing dependencies. Done means the Iceberg input source can read ORC and Avro tables in addition to Parquet.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.