[FEATURE] Support Fileset TTL
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### Describe the feature
In the company, we can use Fileset to manage files in HDFS, and S3. A major requirement is to do lifecycle management on Files. We have experienced a process of data going to the cloud. This process is also the process of users organizing data.
After we completed it, we found that the cost of user data deletion accounted for 50%. This means that having a large amount of unstructured data is useless. Therefore, lifecycle management is very important.
I create this issue to discuss supporting the TTL of Fileset.
### Motivation
Support TTL of Fileset.
### Describe the solution
Here are some simple thoughts.
1. Use the most recent modification time of the file. This solution is more universal but requires to list all files.
2. Use the directory as a date, but this has certain requirements for the path.
### Additional context
_No response_
Contributor guide
Research direction
No implementation files, tests, or entry points are named. Start by locating the Fileset implementation and its HDFS/S3 lifecycle handling, then review the discussion before choosing between modification-time and date-directory approaches. Done should include an agreed TTL design and corresponding behavior for Fileset-managed data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100