Flink Sink V2: Add OutputFileFactoryProvider plugin interface
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
Adds an `OutputFileFactoryProvider` plugin interface to `IcebergSink` that allows external implementations to customize how `OutputFileFactory` instances are created. This enables use cases like custom file naming, path rewriting, or alternative storage routing without subclassing internal writer factories.
### Query Engine
Flink
### Motivation
The current `RowDataTaskWriterFactory` creates `OutputFileFactory` internally with no extension point. Users who need to customize output file paths (e.g., for cross-region writes, custom partitioning schemes, or storage proxy routing) must subclass `RowDataTaskWriterFactory` and depend on its internal structure.
This PR adds a clean plugin interface: if an `OutputFileFactoryProvider` is set on the builder, it is used to create the `OutputFileFactory` in `RowDataTaskWriterFactory.initialize()`. Otherwise the existing behavior is unchanged.
### Changes
- New: `OutputFileFactoryProvider.java` -- `@FunctionalInterface` with a single method: `OutputFileFactory create(Table table, int taskId, int attemptId, FileFormat format, PartitionSpec spec)`
- Modified: `RowDataTaskWriterFactory` -- accepts optional provider, uses it in `initialize()` when present
- Modified: `IcebergSink.Builder` -- new `outputFileFactoryProvider()` method, passed through to writer factory
### Compatibility
- No behavioral change when the provider is not set (null default)
- No changes to public API signatures of existing methods
- Fully backward compatible
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start with the new OutputFileFactoryProvider.java contract, then read RowDataTaskWriterFactory.initialize() and the IcebergSink.Builder changes described in the issue. Trace how the optional provider reaches the writer factory, and verify that a configured provider creates the output factory while the unset case preserves existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100