Add support to validate table's internal state with integ test infra
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
we might need to add validation to integ test suite to test data and metadata table's internal state or steady state. For eg, need to validate that
cleaning has progressed as per the configs.
Archival has progressed as per the configs. archival has archived the timeline files.
if metadata is enabled, metadata table compaction progressed as per the configs.
if metadata is enabled, metadata table cleaning has progressed as per the configs.
if metadata is enabled, metadata table archival has progressed and has archived the timeline files.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4022
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the integration test suite and its existing configuration-based checks for data and metadata tables. Define completion around validating cleaning, archival, timeline-file archival, and, when metadata is enabled, compaction and metadata-table cleaning or archival against the configured progress.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100