apache / apache/hudi

show_logfile_records / show_logfile_metadata crash on common MOR log shapes

Open
#19,634 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem**

Four independent crashes in the `show_logfile_records` / `show_logfile_metadata` procedures:

1. **Uncompacted MOR table + `merge => true`**: the merged scan resolves the latest instant via `getActiveTimeline.getCommitAndReplaceTimeline.lastInstant.get`; a deltacommit-only table has an empty commit timeline, so this throws `NoSuchElementException`. This is the procedure's primary use case (inspecting logs before compaction). The CLI twin was fixed in #13508 with `.map(HoodieInstant::requestedTime).orElse(currentInstantTime)`; the procedure never got the guard.
2. **Non-data log blocks**: both procedures match `case dataBlock: HoodieDataBlock` with no default arm, so a delete block (v9 inline logs, or v10 native `.deletes.*` files) or a rollback command block raises `scala.MatchError`. The CLI original has guarded with `instanceof` since 2020 (#1687); the #5949 port dropped the guard. Observed as `scala.MatchError: ...HoodieDeleteBlock@... (of class org.apache.hudi.common.table.log.block.HoodieDeleteBlock)` from both procedures on a v9 log file holding a data block plus a delete block (one `merge into` with an update and a delete clause).
3. **`merge => true` with a key updated in more than one log block**: the merged record's data is a `SerializableIndexedRecord`, and the unconditional `asInstanceOf[HoodieRecordPayload[_]]` in `ShowHoodieLogFileRecordsProcedure` throws `ClassCastException`.
4. **Log file without a data block in the glob**: both branches of `show_logfile_records` `Objects.requireNonNull` a schema read off a log file and die with a bare `NullPointerException` (no message) when the file has no data block -- but at different sites. The merged branch reads the schema from `logFilePaths.last`, so it dies whenever the LAST globbed file lacks a data block, even if earlier files have one. The unmerged branch re-reads the schema per file inside a `takeWhile(records < limit)` stream, so it dies on the FIRST reached file without a data block -- and may not die at all if `limit` is filled by earlier files. The simplest trigger is a plain `delete from` on a v9 MOR table, which writes the delete block into its own log file. (`show_logfile_metadata` on the same file reaches crash 2 instead.) The CLI twin fixed exactly this in HUDI-6694 (#9445, `7ffcbfbeeba1`): per-file null tolerance plus a backward scan over the log files for the reader schema (`HoodieLogFileCommand.java:119`, `:223-231`), neither of which was ported to the procedure.

**To reproduce**

(1) MOR table, inserts + updates only, `show_logfile_records(..., merge => true)`. (2) v9 MOR table, one `merge into` with a matched update and a matched delete, then point either procedure at the resulting log file. (3) update the same key twice, then `merge => true`. (4) v9 MOR table, plain `delete from`, then `show_logfile_records` over the delete's log file. `TestHoodieLogFileProcedure` (since #19161) pins crashes 2 and 4 and documents the workarounds for 1 and 3 in comments referencing this issue.

**Suggested fix**

Port the CLI's `lastInstant` fallback, add a `case _ =>` skip arm to both matches, handle `HoodieAvroIndexedRecord` / `SerializableIndexedRecord` data in the merged path, and port the HUDI-6694 schema resolution (per-file null tolerance + backward scan) instead of `requireNonNull` on the last file -- null tolerance alone is not enough for the merged path, which needs a schema from whichever file has one.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with TestHoodieLogFileProcedure and compare the procedures with the CLI implementation in HoodieLogFileCommand.java, especially the cited schema-resolution lines. Reproduce the four MOR log shapes, then verify both procedures avoid the crashes while preserving record and metadata inspection for data, delete, rollback, and schema-less log files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.