apache / apache/iceberg

Flink Sink V2: Add OutputFileFactoryProvider plugin interface

Open
#15,763 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Adds an `OutputFileFactoryProvider` plugin interface to `IcebergSink` that allows external implementations to customize how `OutputFileFactory` instances are created. This enables use cases like custom file naming, path rewriting, or alternative storage routing without subclassing internal writer factories.

### Query Engine

Flink

### Motivation

The current `RowDataTaskWriterFactory` creates `OutputFileFactory` internally with no extension point. Users who need to customize output file paths (e.g., for cross-region writes, custom partitioning schemes, or storage proxy routing) must subclass `RowDataTaskWriterFactory` and depend on its internal structure.

This PR adds a clean plugin interface: if an `OutputFileFactoryProvider` is set on the builder, it is used to create the `OutputFileFactory` in `RowDataTaskWriterFactory.initialize()`. Otherwise the existing behavior is unchanged.

### Changes

- New: `OutputFileFactoryProvider.java` -- `@FunctionalInterface` with a single method: `OutputFileFactory create(Table table, int taskId, int attemptId, FileFormat format, PartitionSpec spec)`
- Modified: `RowDataTaskWriterFactory` -- accepts optional provider, uses it in `initialize()` when present
- Modified: `IcebergSink.Builder` -- new `outputFileFactoryProvider()` method, passed through to writer factory

### Compatibility

- No behavioral change when the provider is not set (null default)
- No changes to public API signatures of existing methods
- Fully backward compatible

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with the new OutputFileFactoryProvider.java contract, then read RowDataTaskWriterFactory.initialize() and the IcebergSink.Builder changes described in the issue. Trace how the optional provider reaches the writer factory, and verify that a configured provider creates the output factory while the unset case preserves existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.