Persist totalBytesWritten as part of HoodieCleanMetadata
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
{{Improvements in observability/metrics for Hudi OSS}}
{{1.HoodieCleanMetadata}} contains only the number of files deleted and the name of the files but doesn't contain the total bytes that have been cleaned up. This is an important metric to keep track of and total size in bytes metrics if computed will always be more and will keep missing the bytes removed by the cleaner.
2. In HoodieWriteStat, we don't have a metric to know how much data is being added/removed as part of each file slice being written. If we want to know how many bytes are being added/remove in the table as part of each commit we need this metric as part of HoodieWriteStat.
3. latency fields - totalUpsertTime, totalCreateTime, totalScanTime are missing in archived HoodieCommitMetadata. This makes bootstrapping the metrics for a table having archived commits difficult and not possible unless we have the active commit me
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3788
- Type: Task
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating HoodieCleanMetadata, HoodieWriteStat, and HoodieCommitMetadata, then inspect their serialization and any related tests. Determine how persisted and archived metadata is read before adding the requested byte and latency metrics; done means cleaner, write-stat, and archived-commit metadata expose the measurements described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100