apache / apache/iceberg

Flink Iceberg should support Flink's IOs in addition to Iceberg's IO

Open
#15,352 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

Flink already has a so called FileSystem implementation for S3, HDFS, and others. Iceberg ships its own implementation via FileIO. In many scenarios, Flink's file system is already setup correctly, but still the user has to go through hoops to setup Iceberg's FileIO.

Flink Iceberg should get a wrapper which discovers Flink's FileSystem and defers calls to it.

### Query engine

Flink

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by reading Iceberg's FileIO abstraction and Flink's FileSystem implementation for S3, HDFS, and other filesystems. Trace how Flink Iceberg currently selects and configures FileIO, then determine how a wrapper can discover and delegate to Flink's FileSystem. Done means users with a correctly configured Flink filesystem no longer need separate Iceberg FileIO setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.