apache / apache/hudi

Revisit the design of HoodieRecord metadata

Open
#17,111 1 comment 0 reactions 0 assignees View on GitHub
area:core from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, each HoodieRecord takes an optional Map as a metadata, which is used for function like event_time metadata tracing, but it is not as efficient.

Before write a record in each write handle, we initialize a new map to keep the metadata if the HoodieRecord is empty(now it is always empty), and pass it to the WriteStatus.markSuccess, the write status then fetchs the meta from the map, this happens for every record.

Suggestion, remove the metadata from HoodieRecord, and add API on WriteStauts to collect the whatever metadata we like directly, for e.g,

{code:java}
WriteStatus.collectEventTimeMetadata(Object eventTime).
{code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9659
- Type: Improvement
- Fix version(s):
- 1.2.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing HoodieRecord metadata through each write handle and into WriteStatus.markSuccess, then review how event-time metadata is collected. The issue is done when the metadata path is redesigned as agreed, the per-record map allocation is removed, and existing metadata behavior remains covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.