Report metrics on the number of duplicates after dedup
Open
from-jira
priority:high
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In Delastreamer, user can set --filter-dupes to dedup the input records. Currently, Hudi does not report any metric on the duplicates. It would be good to report such metrics and monitor how many records are filtered in each commit.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5501
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the DeltaStreamer entry point and the --filter-dupes handling to identify where duplicate records are filtered and where per-commit metrics are emitted. Review the existing metric names and tests in those paths; done means reporting how many records are filtered in each commit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100