Warn logs about not able to read replace commit metadata
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I was trying out spark streaming sink w/ hudi and saw warn logs as below.
{code:java}
22/04/09 15:54:16 WARN AbstractTableFileSystemView: Could not read commit details from /tmp/hudi_streaming_kafka/COPY_ON_WRITE/.hoodie/20220409154917240.replacecommit
22/04/09 15:54:16 WARN AbstractTableFileSystemView: Could not read commit details from /tmp/hudi_streaming_kafka/COPY_ON_WRITE/.hoodie/20220409155011647.replacecommit {code}
But ran some validations and ensured data was intact. Further investigation revealed that, this happens just after archival, where in the replace commit shown above were part of the list of instants that got archived. So, may be active timeline reloading is missed somewhere. Since its a warn log and does not cause any correctness issue, filing a low priority ticket.
Steps to repo:
spark streaming write to Hudi COW table w/ async clustering. make archival aggressive and you should see these logs at some point
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3840
- Type: Task
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Spark streaming write to a Hudi copy-on-write table with async clustering and aggressive archival. Trace the warning from AbstractTableFileSystemView for archived .replacecommit files and investigate active timeline reloading. Done means the archival scenario no longer emits the misleading warning while data remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100