Add ORC and Avro file format support to Druid's Iceberg input source
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
Component: extensions-contrib/druid-iceberg-extensions
Druid's Iceberg input source (druid-iceberg-extensions) currently only supports reading Iceberg tables stored in Parquet format.
IcebergNativeRecordReader hardcodes Parquet.read() + GenericParquetReaders for all reads:
### Motivation
This was flaged while working on v2 spec support https://github.com/apache/druid/pull/19266#discussion_r3259155168
dependecies : iceberg-orc, orc-core, iceberg-avro, and avro are absent
### References:
• IcebergNativeRecordReader.java — current Parquet-only implementation
• IcebergFileTaskInputSource.java — serialisation boundary between coordinator and worker
• Iceberg API: org.apache.iceberg.data.GenericDeleteFilter (public, already on classpath)
• Iceberg API: org.apache.iceberg.data.orc.GenericOrcReader, org.apache.iceberg.data.avro.GenericAvroReader
• PR #19266 — added Iceberg V2 delete support (Parquet only); this is the follow-up
Contributor guide
Research direction
Start with extensions-contrib/druid-iceberg-extensions/IcebergNativeRecordReader.java to trace the Parquet-only read path, then inspect IcebergFileTaskInputSource.java for the coordinator-to-worker serialization boundary. Review the referenced GenericOrcReader and GenericAvroReader APIs and the missing dependencies. Done means the Iceberg input source can read ORC and Avro tables in addition to Parquet.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100