apache / apache/iceberg

Decouple Iceberg from Hadoop codebase

Open
#14,284 2 comments 12 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Hi, this is a proposal from the Trino community to decouple Iceberg from the Hadoop codebase.

As some of you may already know, Trino decoupled its Hadoop codebase last year (see trinodb/trino#15921).
The motivation and background for that change are explained in detail in the link above.

We’ve found that Iceberg’s current dependency on Hadoop makes it harder to adopt new Iceberg features.
In many cases, Trino Iceberg connector developers need to reinvent the wheel when a feature depends on the Parquet or ORC reader/writer.
While this is technically possible, it often requires non-trivial effort and increases the risk of bugs or inconsistencies, which isn’t ideal for users.

Since Parquet and ORC are just file formats, relying on Hadoop shouldn’t be a strict requirement.
We’d like to propose supporting native readers and writers for Parquet and ORC files - similar to Trino’s approach which could simplify maintenance and make it easier for downstream projects to stay up to date with Iceberg’s latest features.

Many Trino users expect new Iceberg features to become available shortly after we upgrade the library, and reducing this dependency would help make that possible.

We’d love to hear your thoughts and feedback on this idea.

### Query engine

Trino

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

The issue names no Iceberg files, tests, or entry points. Start by reviewing the referenced Trino decoupling change and mapping Iceberg’s Hadoop dependency, then work with the community to define the scope for native Parquet and ORC readers and writers and how compatibility will be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.