DynamicIcebergSink silently stops committing after ungraceful restart (crash recovery from checkpoint)
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
**Component: Flink / DynamicIcebergSink Versions:**
- Apache Iceberg: 1.11.0 (iceberg-flink-runtime-2.1)
- Apache Flink: 2.2.1
- Flink Kubernetes Operator: 1.15.0
- Catalog: S3 Tables (AWS)
**Description:**
After an ungraceful restart caused by node eviction (EKS node group recycle / AMI rotation), the DynamicIcebergSink enters a state where it:
- Consumes records from Kafka at full rate (~600–2500 records/sec)
- Produces zero Iceberg commits to any table
- Logs no errors or warnings
- Completes checkpoints successfully with zero failures
- Reports zero pendingCommittables and zero successfulCommittables
The job appears completely healthy from the outside (RUNNING state, Kafka offsets advancing, lag ~0, checkpoints completing) but silently writes nothing.
The condition is sticky across internal Flink task-level failovers (restoring from checkpoint reproduces the same broken state). It is only cleared by a full job resubmission that discards all operator state (operator-managed stateless restart or full delete + redeploy).
**Root Cause Hypothesis:**
When the job restores from a checkpoint after an ungraceful crash, the DynamicIcebergSink's internal writer state is restored into a broken or inconsistent state. The writer silently accepts incoming DynamicRecords but never flushes them to data files, resulting in zero committables reaching the committer.
This is distinct from a committer-level failure — the committer is not stuck, it simply has nothing to commit because the writer never produces output.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with DynamicIcebergSink checkpoint restoration and its internal writer state, comparing a task-level failover restore with a full job resubmission. Reproduce the node-eviction scenario and trace whether restored writers produce data files and committables; done means records resume producing Iceberg commits after checkpoint recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100