Add reconcile commit metadata support for flink and java
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
When we added per log file marker, we had to reconcile commit metadata to account for missing log files.
[https://github.com/apache/hudi/blob/492baaf77378a5b2519c55ab9889ed041d943a2a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java#L277]
We added support only for spark. Filing this ticket to fix it for flink and java.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8181
- Type: Improvement
- Fix version(s):
- 0.16.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the referenced reconcile-commit-metadata logic in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java around line 277. Compare the existing Spark support with the Flink and Java paths to identify where missing log files must be accounted for. Done means reconcile commit metadata works for both Flink and Java, with coverage for those paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100