apache / apache/hudi

Archived clean instants break all clean procedures: show_clean_plans returns all-null rows, show_cleans / show_cleans_metadata throw IOException

Open
#19,639 2 comments 0 reactions 1 assignee Claimed by @KiteSoar View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem**

`show_clean_plans(showArchived => true)` returns all-null rows for archived clean plans. `getCleanerPlans` (`ShowCleansPlanProcedure.scala:185-202`) collects archived clean instants from `metaClient.getArchivedTimeline`, but then hands every instant to `processCleanPlan(metaClient, metaClient.getActiveTimeline, cleanInstant)`. Reading the plan of an archived instant off the active timeline fails; the failure is swallowed and `createErrorRow` emits a row whose columns are all null.

The sibling procedures are broken too, just differently: `show_cleans` and `show_cleans_metadata` do route archived instants through `metaClient.getArchivedTimeline` (`ShowCleansProcedure.scala:167-172`), but reading the clean metadata off the archived instant fails Avro magic-byte parsing, so the WHOLE call throws `java.io.IOException: unable to read commit metadata for instant [...clean...]` instead of returning rows. So all three clean procedures mishandle archived cleans: `show_clean_plans` silently returns null-filled rows, the two siblings throw.

The pre-existing `showArchived` test never triggered archival (too few commits), so its `allPlans.length >= activePlans.length` assertion passed at equality and none of this was noticed.

**To reproduce**

Configure aggressive archival (low `hoodie.keep.min.commits` / `hoodie.keep.max.commits`), produce several clean instants, let the timeline archive them, then:
- `call show_clean_plans(table => 't', showArchived => true)`: the archived rows come back with null `policy`, `earliest_instant_to_retain`, etc.
- `call show_cleans(table => 't', showArchived => true)` or `show_cleans_metadata(...)`: throws `IOException: unable to read commit metadata for instant`.

**Suggested fix**

Root cause of the sibling throw: `metaClient.getArchivedTimeline()` loads instants with `LoadMode.ACTION`, which never loads instant content -- `getInstantDetails` returns an empty stream, so `readCleanMetadata` fails with the IOException above. The bytes are in the archive; they were just never loaded. So the sibling fix is a load-mode change: call `loadCompletedInstantDetailsInMemory()` (LoadMode.METADATA) or use the `getArchivedTimeline(startTs)` overload before reading. `show_clean_plans` additionally needs the PLAN content, for which no ready helper exists (`loadCompactionDetailsInMemory` filters to commit actions only). Merely passing the archived timeline the way `ShowCleansProcedure` does is NOT sufficient -- that path throws, as above. Regression coverage: an archival-forcing test in `TestShowCleansProcedures` (since #19161) pins today's behavior for all three procedures -- null plan fields from `show_clean_plans`, `IOException` from the siblings -- and flips when fixed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.