Make test datagen ensure records in specified partitions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
For some test cases we generate 100 or 1000 record to ensure there are records in all 3 fixed partitions. This wastes a lot of time in running the tests. UT or FT should not need to test against many records. Usually 10-20 records with parallelism of 2 will suffice.
We should audit the current datagen usage through all test cases and optimize in this aspect.
Tests with large number of records should be moved to long-running integration tests. (this is for another story)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5287
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
Contributor guide
No contributing guide indexed for this repository
Research direction
Search all test cases for datagen usage and record-count settings, then identify UT and FT cases generating 100 or 1000 records for the three fixed partitions. Reduce suitable cases toward 10–20 records with parallelism 2, verify each still populates every partition, and leave large-record integration-test migration for the separate story.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100