apache / apache/hudi

[FEATURE REQUEST] Archival support for metadata table of Record level indexing

Open
#11,195 3 comments 0 reactions 0 assignees View on GitHub
area:metadata-table type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hi Community,

I am seeking guidance on implementing archival policies for the Metadata table in Hudi's record-level indexing (RLI). We are not concerned with retaining data older than three years, as we move that data from standard S3 storage to Glacier. Additionally, we would like to archive the metadata record keys provided by Hudi's record-level indexing to keep storage costs under control. I wanted to inquire if there are any plans to introduce TTL or Archival support for the Hudi metadata table's record keys. Is there any plan to bring this feature.

Thanks

cc - @codope

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name a file, test, or entry point. Start by reviewing Hudi's record-level indexing metadata table and existing retention or archival behavior, then clarify the intended TTL scope, Glacier interaction, and completion criteria before implementation.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.