apache / apache/hudi

Meta-sync completion-time watermark has no test coverage and the Glue test fixture commit is invisible to the timeline

Open
#19,725 1 comment 0 reactions 1 assignee Claimed by @Davis-Zhang-Onehouse View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

HUDI-7494 (#10841) taught the meta-sync clients to persist and read `last_commit_completion_time_sync`, so incremental sync can register a commit that completes after the instant-time watermark has already moved past it (for example a long-running INSERT_OVERWRITE overtaken by a later-instant commit). If that mechanism regresses, the affected partition is silently and permanently dropped from the catalog: every later cycle either short-circuits on `isAlreadySynced` or resolves zero partitions and advances the watermark again. Nothing in the test suites fails today if the completion watermark stops being written, is computed with the wrong ordering, or is never consulted on read.

Separately, `GlueTestUtil.createHoodieTable` writes its fixture commit into `.hoodie/`, while the version-8 timeline reads `.hoodie/timeline/`, so `TestAWSGlueSyncClient` runs every test against an empty active timeline without noticing.

Both are addressed by adding sync-client coverage: falsifiable unit tests on `AWSGlueCatalogSyncClient`, an end-to-end `HiveSyncTool` test of the hollow-instant rescue, and a fixture that actually lands in the timeline folder.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.