S3/GCS IncrSource drops records at mid-commit pagination checkpoint
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
The S3/GCS cloud-object incremental sources (`S3EventsHoodieIncrSource` / `GcsEventsHoodieIncrSource`) can silently drop records when a previous batch persisted a `commit#fileKey` mid-commit-pagination checkpoint (i.e. the prior batch hit `sourceLimit` before finishing the start commit's files). Files in the start commit after the checkpoint key become unreachable, and the checkpoint advances past them.
Root causes:
1. `QueryRunner.runIncrementalQuery` passes `queryInfo.getStartInstant()` as the Spark `START_COMMIT`. The V1 incremental relation's range is start-exclusive, so the start commit is dropped from the scan; the downstream `(commit_time || object_key) > 'commit#fileKey'` filter then matches nothing in the start commit and the checkpoint is emitted as a bare instant, skipping the remaining files.
2. The file-group-reader based read path drops rows when a filter references a column not in the required schema (the required filters from the relation factory are not visible to the Spark planner, so the required schema can miss filter columns).
Triggers: cold-start backfills against a source table with a large initial commit, bursty writers, or a low `hoodie.deltastreamer.read.source.limit`.
### Expected
The incremental query includes the start commit so mid-commit pagination resumes correctly and no records are dropped.
### Related
apache/hudi #18949 proposed this fix against an older master and is stale/conflicting. This is a fresh version rebased onto current master.
### Environment
- Hudi version: master (1.x)
- S3/GCS incremental sources with source-limit based batching.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.