Benchmark MDT w/ 1000 commits and cleaner set to 800
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Essentially we want to benchmarking cleaner along w/ MDT. when there are 800+ commits in active timeline, how does MDT performs.
1000 partitions.
each commit updates random 5% of partitions.
1000+ commits.
cleaner commits retained = 800.
Triage any performance hit on the write path.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5439
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Locate the existing MDT and cleaner benchmark entry points, then run the stated workload: 1,000 partitions, random 5% updates per commit, 1,000+ commits, and 800 retained cleaner commits. Compare the write path with and without cleaner activity and record any performance impact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100