[RFC-87] RowData log handle supports writing record positions to log block
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Flink writer do not support record-position for updates and deletes yet, will support it later.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9192
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9075
- Fix version(s):
- 1.2.0
---
## Comments
30/Apr/25 09:09;geserdugarov;I've raised more questions then figured out how to implement it properly during research of this task.
Initially, Flink integration doesn't support record positions processing even for `HoodieRecord`s. But I didn't found any explicit description of how "log records positions" feature should work. So, we have to use Spark implementation as an example.
Writing of log records positions and using them are separated between different PRs, and were implemented by different developers:
1) additional property in log block header (without writing): [https://github.com/apache/hudi/pull/9376]
2) writing log record positions in log block header: [https://github.com/apache/hudi/pull/9581]
3) using of log record positions from log block headers in file group readers: [https://github.com/apache/hudi/pull/9819]
Confusing part here, that we write log record positions as a set in `HoodieLogBlock::addRecordPositionsIfRequired`:
[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java#L390-L392]
But later extract them as a list in `PositionBasedFileGroupRecordBuffer::extractRecordPositions`:
[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java#L305-L307]
and use them as a list in `PositionBasedFileGroupRecordBuffer::processDataBlock`:
[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java#L132-L136]
;;;
---
30/Apr/25 11:51;geserdugarov;During checking of cases when log record positions are written, found another issue with not persistent index configuration:
https://github.com/apache/hudi/issues/13241;;;
---
30/Apr/25 12:03;geserdugarov;Need to clarify a lot of different questions first. So I postponed this task for now, and unassigned it from myself to allow anybody to work on this task.;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked Spark PRs 9376, 9581, and 9819, then inspect HoodieLogBlock::addRecordPositionsIfRequired and PositionBasedFileGroupRecordBuffer::extractRecordPositions and processDataBlock. Clarify the record-position requirements for Flink, including the related persistent-index issue 13241; done means Flink writers support record positions for updates and deletes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100