apache / apache/hudi

S3/GCS IncrSource drops records at mid-commit pagination checkpoint

Open
#19,756 0 comments 0 reactions 1 assignee Claimed by @lokeshj1703 View on GitHub
area:deltastreamer priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

The S3/GCS cloud-object incremental sources (`S3EventsHoodieIncrSource` / `GcsEventsHoodieIncrSource`) can silently drop records when a previous batch persisted a `commit#fileKey` mid-commit-pagination checkpoint (i.e. the prior batch hit `sourceLimit` before finishing the start commit's files). Files in the start commit after the checkpoint key become unreachable, and the checkpoint advances past them.

Root causes:
1. `QueryRunner.runIncrementalQuery` passes `queryInfo.getStartInstant()` as the Spark `START_COMMIT`. The V1 incremental relation's range is start-exclusive, so the start commit is dropped from the scan; the downstream `(commit_time || object_key) > 'commit#fileKey'` filter then matches nothing in the start commit and the checkpoint is emitted as a bare instant, skipping the remaining files.
2. The file-group-reader based read path drops rows when a filter references a column not in the required schema (the required filters from the relation factory are not visible to the Spark planner, so the required schema can miss filter columns).

Triggers: cold-start backfills against a source table with a large initial commit, bursty writers, or a low `hoodie.deltastreamer.read.source.limit`.

### Expected

The incremental query includes the start commit so mid-commit pagination resumes correctly and no records are dropped.

### Related

apache/hudi #18949 proposed this fix against an older master and is stale/conflicting. This is a fresh version rebased onto current master.

### Environment

- Hudi version: master (1.x)
- S3/GCS incremental sources with source-limit based batching.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.