apache / apache/hudi

Metadata table validator loads the entire archived timeline and misclassifies the range lower bound

Open
#19,625 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:core area:tools priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem you faced

`HoodieMetadataTableValidator.computeDiffSummary` builds the diagnostic message for a detected file-slice mismatch. To decide whether the file slices it found only on storage belong to commits that were archived (so the metadata table is genuinely missing a committed file) or to commits that exist nowhere (so they are orphans of a failed or rolled-back write), it consults the archived timeline:

```java
Set archivedInstants = metaClient.getArchivedTimeline()
.findInstantsInRange(minInstant, maxInstant)
.getInstantsAsStream()
.map(HoodieInstant::requestedTime)
.collect(Collectors.toSet());
```

Two problems follow from filtering in memory rather than pushing the range into the load.

**1. The whole archive is loaded.** `getArchivedTimeline()` is the no-arg accessor, so every archived instant is materialized with its commit metadata, and the result is cached in `archivedTimelineMap` for the rest of the task. On a large table this is enough to OOM an executor. Because the load happens while the error message is being built, the `HoodieValidationException` carrying the mismatch verdict is never thrown: the validator reports an unexplained failure and the real finding, a genuine file-slice count mismatch, is lost. "No mismatch reported for this table" is therefore not evidence the table is clean.

Note that a range filter alone would not fix the memory use. `ArchivedTimelineLoaderV1` applies the filter only after deserializing each record, and the reverse-chronological early `break` that skips older archive files is disabled entirely when the filter is null, which is exactly today's path.

**2. An off-by-one on the lower bound.** `findInstantsInRange` is half-open: `InstantComparison.isInRange` is `GREATER_THAN startTs` and `LESSER_THAN_OR_EQUALS endTs`. `minInstant` is itself a member of the set being classified, so it can never appear in `archivedInstants` and always survives into `missingCommits`. When the set has a single element, `minInstant` equals `maxInstant`, the range is empty, and that instant is always reported missing. Beyond the wrong message, a spurious `missingCommits` triggers a scan that reads every `.rollback.requested` plan from storage.

### To Reproduce

Run the metadata table validator against a table with an archived timeline and at least one file slice on storage whose base instant is not in the active timeline. The instant equal to the lower bound of the computed range is always classified as a missing commit even when it is archived.

### Expected behavior

The archive lookup should load only the instants in the range it cares about, and should treat both bounds as inclusive so an archived instant equal to the lower bound is recognized as archived. Separately, a failure while computing a diagnostic summary should degrade the message rather than discard the mismatch verdict the validator had already reached.

### Environment Description

* Hudi version : 1.1.0 and later, including master

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

### Additional context

`TimelineFactory` exposes only `createArchivedTimeline(metaClient)` and `createArchivedTimeline(metaClient, startTs)`, and `HoodieTableMetaClient` correspondingly has no closed-range accessor, which is why the range ended up being applied in memory. `ArchivedTimelineV1` already has a public constructor taking `(metaClient, startTs, endTs)` backed by `ClosedClosedTimeRangeFilter`; it is simply unreachable through the sanctioned API.

`getArchivedTimeline(minInstant)` is not a workaround: `StartTsFilter` is open-ended upward, so it still reads nearly every archive file when `minInstant` is old, and it loads instant details.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.