apache / apache/hudi

Persist totalBytesWritten as part of HoodieCleanMetadata

Open
#15,105 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

{{Improvements in observability/metrics for Hudi OSS}}

{{1.HoodieCleanMetadata}} contains only the number of files deleted and the name of the files but doesn't contain the total bytes that have been cleaned up. This is an important metric to keep track of and total size in bytes metrics if computed will always be more and will keep missing the bytes removed by the cleaner. 
2. In HoodieWriteStat, we don't have a metric to know how much data is being added/removed as part of each file slice being written. If we want to know how many bytes are being added/remove in the table as part of each commit we need this metric as part of HoodieWriteStat. 
3. latency fields - totalUpsertTime, totalCreateTime, totalScanTime are missing in archived HoodieCommitMetadata. This makes bootstrapping the metrics for a table having archived commits difficult and not possible unless we have the active commit me  

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3788
- Type: Task

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating HoodieCleanMetadata, HoodieWriteStat, and HoodieCommitMetadata, then inspect their serialization and any related tests. Determine how persisted and archived metadata is read before adding the requested byte and latency metrics; done means cleaner, write-stat, and archived-commit metadata expose the measurements described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.