Revisit the design of HoodieRecord metadata
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, each HoodieRecord takes an optional Map as a metadata, which is used for function like event_time metadata tracing, but it is not as efficient.
Before write a record in each write handle, we initialize a new map to keep the metadata if the HoodieRecord is empty(now it is always empty), and pass it to the WriteStatus.markSuccess, the write status then fetchs the meta from the map, this happens for every record.
Suggestion, remove the metadata from HoodieRecord, and add API on WriteStauts to collect the whatever metadata we like directly, for e.g,
{code:java}
WriteStatus.collectEventTimeMetadata(Object eventTime).
{code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9659
- Type: Improvement
- Fix version(s):
- 1.2.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing HoodieRecord metadata through each write handle and into WriteStatus.markSuccess, then review how event-time metadata is collected. The issue is done when the metadata path is redesigned as agreed, the per-record map allocation is removed, and existing metadata behavior remains covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100