[FEATURE REQUEST] Archival support for metadata table of Record level indexing
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Hi Community,
I am seeking guidance on implementing archival policies for the Metadata table in Hudi's record-level indexing (RLI). We are not concerned with retaining data older than three years, as we move that data from standard S3 storage to Glacier. Additionally, we would like to archive the metadata record keys provided by Hudi's record-level indexing to keep storage costs under control. I wanted to inquire if there are any plans to introduce TTL or Archival support for the Hudi metadata table's record keys. Is there any plan to bring this feature.
Thanks
cc - @codope
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue does not name a file, test, or entry point. Start by reviewing Hudi's record-level indexing metadata table and existing retention or archival behavior, then clarify the intended TTL scope, Glacier interaction, and completion criteria before implementation.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100