apache / apache/iceberg

DynamicIcebergSink silently stops committing after ungraceful restart (crash recovery from checkpoint)

Open
#17,494 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Flink

### Please describe the bug 🐞

**Component: Flink / DynamicIcebergSink Versions:**
- Apache Iceberg: 1.11.0 (iceberg-flink-runtime-2.1)
- Apache Flink: 2.2.1
- Flink Kubernetes Operator: 1.15.0
- Catalog: S3 Tables (AWS)

**Description:**
After an ungraceful restart caused by node eviction (EKS node group recycle / AMI rotation), the DynamicIcebergSink enters a state where it:
- Consumes records from Kafka at full rate (~600–2500 records/sec)
- Produces zero Iceberg commits to any table
- Logs no errors or warnings
- Completes checkpoints successfully with zero failures
- Reports zero pendingCommittables and zero successfulCommittables

The job appears completely healthy from the outside (RUNNING state, Kafka offsets advancing, lag ~0, checkpoints completing) but silently writes nothing.

The condition is sticky across internal Flink task-level failovers (restoring from checkpoint reproduces the same broken state). It is only cleared by a full job resubmission that discards all operator state (operator-managed stateless restart or full delete + redeploy).

**Root Cause Hypothesis:**
When the job restores from a checkpoint after an ungraceful crash, the DynamicIcebergSink's internal writer state is restored into a broken or inconsistent state. The writer silently accepts incoming DynamicRecords but never flushes them to data files, resulting in zero committables reaching the committer.

This is distinct from a committer-level failure — the committer is not stuck, it simply has nothing to commit because the writer never produces output.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with DynamicIcebergSink checkpoint restoration and its internal writer state, comparing a task-level failover restore with a full job resubmission. Reproduce the node-eviction scenario and trace whether restored writers produce data files and committables; done means records resume producing Iceberg commits after checkpoint recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.