Metadata table does not respect data table's clean and archival settings
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem you faced
When `hoodie.clean.automatic` or `hoodie.archive.automatic` are set to false on the data table, the metadata table (MDT) does not respect these settings and continues to perform archival and clean operations. This is inconsistent behavior since MDT should follow the data table's configuration.
### Describe the solution you'd like
When cleans and archivals are disabled on the data table, these operations should also be disabled on the metadata table. The MDT should check `dataWriteConfig.isAutoClean()` and `dataWriteConfig.isAutoArchive()` before running the corresponding operations.
### Describe alternatives you've considered
None - the MDT should respect the parent table's configuration settings.
### Additional context
Implementation PR: #18145
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the metadata table's clean and archival operations and inspect how dataWriteConfig.isAutoClean() and dataWriteConfig.isAutoArchive() are used. The change is complete when the metadata table skips each operation whenever the corresponding setting is disabled on the data table; review implementation PR #18145 for current work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100