apache / apache/iceberg

Spark streaming: several micro-batch bugs on frequently-compacted sources

Open
#17,001 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.4.

### Query engine

Spark

### Please describe the bug 🐞

When streaming from a table that's compacted (appends interleaved with replace snapshots, `streaming-skip-overwrite-snapshots=true`), each micro-batch reads fewer rows than the configured limit allows, and a restart can produce duplicate rows.

While working on a large scale streaming job we found a few related bugs in the sync/async micro batch planners (spark/v3.5):

1. Skipped snapshots count against the limit. Files in a replace snapshot are counted against the limit, but then planning skips them. The batch ends up smaller than the limit allows.

2. When row limit is unset the value is treated as Integer.MAX_VALUE. Setting only a file limit caps each microbatch at ~2,147,483,647 rows. The counters are also int which causes overflows for large row numbers.

3. The async planner fills treats the start snapshot differently in `fillQueue(Snapshot readFrom)` and `fillQueue(fromOffset, toOffset)`. The read from version applies shouldProcess and skips replace/overwrite/delete. The from offset enqueues the start snapshot. This could be a compaction snapshot so re-reads the rewritten files. In practice the batch planner will not commit an offset inside a non-append snapshot, so this is hard to hit. But it is better to apply the same skip check defensively.

Expected behavior
* Only append snapshots contribute to the rows/files limit. replace/overwrite/delete are skipped.
* A single micro-batch can span multiple appends and cross intervening replace snapshots to reach the configured streaming-max-files-per-micro-batch / streaming-max-rows-per-micro-batch, counting only appended rows.
* A files-only or unbounded row limit is genuinely unbounded.
* A restart never re-reads already-delivered rows.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in the Spark v3.5 sync and async micro-batch planners, focusing on fillQueue(Snapshot readFrom) and fillQueue(fromOffset, toOffset). Trace how shouldProcess, snapshot skipping, row/file counters, and configured limits affect each batch. Done means append-only counting reaches the configured limits, unbounded limits do not overflow, and restarts do not re-read delivered rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.