apache / apache/iceberg

Flink Sink V2: Add PostCommitHook plugin interface

Open
#15,768 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Adds a `PostCommitHook` plugin interface to `IcebergSink` that is invoked after each successful Iceberg commit. This enables use cases like updating table properties, triggering notifications, recording metrics, or synchronizing external metadata stores.

### Query Engine

Flink

### Motivation

After a successful Iceberg commit, users often need to perform additional operations (e.g., write watermark metadata to table properties, notify downstream systems, update external catalogs). Currently there is no extension point -- users must subclass `IcebergCommitter` and override `commit()`, which requires forwarding many constructor arguments through the subclass chain.

This adds a `PostCommitHook` that is called after `operation.commit()` with the snapshot ID and snapshot summary. The hook runs in the committer's context and has access to all commit metadata.

### Changes

- New: `PostCommitHook.java` -- `@FunctionalInterface` with a single method: `void afterCommit(long snapshotId, Map summary)`
- Modified: `IcebergCommitter` -- accepts optional hook, invokes it after successful commit
- Modified: `IcebergSink.Builder` -- new `postCommitHook()` method, passed through to committer

### Compatibility

- No behavioral change when the hook is not set (null default)
- No changes to public API signatures of existing methods
- Fully backward compatible

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with the new PostCommitHook.java interface, then trace IcebergCommitter's successful operation.commit() flow and IcebergSink.Builder's option handling. Add the hook plumbing described in the issue, and verify that it receives the snapshot ID and summary after commits while behavior remains unchanged when unset.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.