apache / apache/hudi

Warn logs about not able to read replace commit metadata

Open
#15,117 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:clustering engine:spark from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I was trying out spark streaming sink w/ hudi and saw warn logs as below. 
{code:java}
22/04/09 15:54:16 WARN AbstractTableFileSystemView: Could not read commit details from /tmp/hudi_streaming_kafka/COPY_ON_WRITE/.hoodie/20220409154917240.replacecommit
22/04/09 15:54:16 WARN AbstractTableFileSystemView: Could not read commit details from /tmp/hudi_streaming_kafka/COPY_ON_WRITE/.hoodie/20220409155011647.replacecommit {code}
But ran some validations and ensured data was intact. Further investigation revealed that, this happens just after archival, where in the replace commit shown above were part of the list of instants that got archived. So, may be active timeline reloading is missed somewhere. Since its a warn log and does not cause any correctness issue, filing a low priority ticket. 

 

Steps to repo:

spark streaming write to Hudi COW table w/ async clustering. make archival aggressive and you should see these logs at some point

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3840
- Type: Task

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Spark streaming write to a Hudi copy-on-write table with async clustering and aggressive archival. Trace the warning from AbstractTableFileSystemView for archived .replacecommit files and investigate active timeline reloading. Done means the archival scenario no longer emits the misleading warning while data remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.