apache / apache/hudi

Metadata table does not respect data table's clean and archival settings

Open
#18,150 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem you faced

When `hoodie.clean.automatic` or `hoodie.archive.automatic` are set to false on the data table, the metadata table (MDT) does not respect these settings and continues to perform archival and clean operations. This is inconsistent behavior since MDT should follow the data table's configuration.

### Describe the solution you'd like

When cleans and archivals are disabled on the data table, these operations should also be disabled on the metadata table. The MDT should check `dataWriteConfig.isAutoClean()` and `dataWriteConfig.isAutoArchive()` before running the corresponding operations.

### Describe alternatives you've considered

None - the MDT should respect the parent table's configuration settings.

### Additional context

Implementation PR: #18145

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the metadata table's clean and archival operations and inspect how dataWriteConfig.isAutoClean() and dataWriteConfig.isAutoArchive() are used. The change is complete when the metadata table skips each operation whenever the corresponding setting is disabled on the data table; review implementation PR #18145 for current work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.