apache / apache/hudi

[RFC-87] RowData log handle supports writing record positions to log block

Open
#17,408 2 comments 0 reactions 0 assignees View on GitHub
area:sql from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Flink writer do not support record-position for updates and deletes yet, will support it later.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9192
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9075
- Fix version(s):
- 1.2.0

---

## Comments

30/Apr/25 09:09;geserdugarov;I've raised more questions then figured out how to implement it properly during research of this task.

 

Initially, Flink integration doesn't support record positions processing even for `HoodieRecord`s. But I didn't found any explicit description of how "log records positions" feature should work. So, we have to use Spark implementation as an example.

Writing of log records positions and using them are separated between different PRs, and were implemented by different developers:

1) additional property in log block header (without writing): [https://github.com/apache/hudi/pull/9376]

2) writing log record positions in log block header: [https://github.com/apache/hudi/pull/9581]

3) using of log record positions from log block headers in file group readers: [https://github.com/apache/hudi/pull/9819]

 

Confusing part here, that we write log record positions as a set in `HoodieLogBlock::addRecordPositionsIfRequired`:

[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java#L390-L392]

But later extract them as a list in `PositionBasedFileGroupRecordBuffer::extractRecordPositions`:

[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java#L305-L307]

and use them as a list in `PositionBasedFileGroupRecordBuffer::processDataBlock`:

[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java#L132-L136]

 ;;;

---

30/Apr/25 11:51;geserdugarov;During checking of cases when log record positions are written, found another issue with not persistent index configuration:

https://github.com/apache/hudi/issues/13241;;;

---

30/Apr/25 12:03;geserdugarov;Need to clarify a lot of different questions first. So I postponed this task for now, and unassigned it from myself to allow anybody to work on this task.;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked Spark PRs 9376, 9581, and 9819, then inspect HoodieLogBlock::addRecordPositionsIfRequired and PositionBasedFileGroupRecordBuffer::extractRecordPositions and processDataBlock. Clarify the record-position requirements for Flink, including the related persistent-index issue 13241; done means Flink writers support record positions for updates and deletes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.