Rename "Deleted Segments" feature to "Archived Segments"
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Proposing a somewhat cosmetic change and some related small features.
Pinot already has a mechanism for keeping 'deleted' segments around for a configurable period under `/Deleted_segments` directory.
I'm proposing to rename this feature from _Deleted Segments_ to _Archived Segments_ and provide more configuration options:
- Retention period (Already exists)
- Archival location (not hard coded to '/Deleted_segments')
- Archival format (Capability to use different storage formats, e.g. psf, parquet, etc.)
- ...(Open to more ideas)
This would make the archival feature more useful for recovery purposes. Users can specify external locations with different durability and quota properties. With customized archival formats, users can optimize based on their needs, data size, query-ability etc.
I'm willing to get the implementation going if I can get some alignment on the idea. Please let me know what you think.
cc @chenboat @ankitsultana @Jackie-Jiang
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the existing Deleted_segments retention mechanism and clarify which proposed changes are in scope: the rename, archival location, and archival format options. Done requires an agreed design and implementation scope rather than an open-ended set of archival ideas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100