apache / apache/gravitino

[FEATURE] Support Fileset TTL

Open
#2,372 10 comments 1 reaction 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### Describe the feature

In the company, we can use Fileset to manage files in HDFS, and S3. A major requirement is to do lifecycle management on Files. We have experienced a process of data going to the cloud. This process is also the process of users organizing data.
After we completed it, we found that the cost of user data deletion accounted for 50%. This means that having a large amount of unstructured data is useless. Therefore, lifecycle management is very important.

I create this issue to discuss supporting the TTL of Fileset.

### Motivation

Support TTL of Fileset.

### Describe the solution

Here are some simple thoughts.
1. Use the most recent modification time of the file. This solution is more universal but requires to list all files.
2. Use the directory as a date, but this has certain requirements for the path.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No implementation files, tests, or entry points are named. Start by locating the Fileset implementation and its HDFS/S3 lifecycle handling, then review the discussion before choosing between modification-time and date-directory approaches. Done should include an agreed TTL design and corresponding behavior for Fileset-managed data.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.