Decouple Iceberg from Hadoop codebase
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
Hi, this is a proposal from the Trino community to decouple Iceberg from the Hadoop codebase.
As some of you may already know, Trino decoupled its Hadoop codebase last year (see trinodb/trino#15921).
The motivation and background for that change are explained in detail in the link above.
We’ve found that Iceberg’s current dependency on Hadoop makes it harder to adopt new Iceberg features.
In many cases, Trino Iceberg connector developers need to reinvent the wheel when a feature depends on the Parquet or ORC reader/writer.
While this is technically possible, it often requires non-trivial effort and increases the risk of bugs or inconsistencies, which isn’t ideal for users.
Since Parquet and ORC are just file formats, relying on Hadoop shouldn’t be a strict requirement.
We’d like to propose supporting native readers and writers for Parquet and ORC files - similar to Trino’s approach which could simplify maintenance and make it easier for downstream projects to stay up to date with Iceberg’s latest features.
Many Trino users expect new Iceberg features to become available shortly after we upgrade the library, and reducing this dependency would help make that possible.
We’d love to hear your thoughts and feedback on this idea.
### Query engine
Trino
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
The issue names no Iceberg files, tests, or entry points. Start by reviewing the referenced Trino decoupling change and mapping Iceberg’s Hadoop dependency, then work with the community to define the scope for native Parquet and ORC readers and writers and how compatibility will be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100