apache / apache/pinot

Rename "Deleted Segments" feature to "Archived Segments"

Open
#12,424 4 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Proposing a somewhat cosmetic change and some related small features.

Pinot already has a mechanism for keeping 'deleted' segments around for a configurable period under `/Deleted_segments` directory.

I'm proposing to rename this feature from _Deleted Segments_ to _Archived Segments_ and provide more configuration options:
- Retention period (Already exists)
- Archival location (not hard coded to '/Deleted_segments')
- Archival format (Capability to use different storage formats, e.g. psf, parquet, etc.)
- ...(Open to more ideas)

This would make the archival feature more useful for recovery purposes. Users can specify external locations with different durability and quota properties. With customized archival formats, users can optimize based on their needs, data size, query-ability etc.

I'm willing to get the implementation going if I can get some alignment on the idea. Please let me know what you think.

cc @chenboat @ankitsultana @Jackie-Jiang

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the existing Deleted_segments retention mechanism and clarify which proposed changes are in scope: the rename, archival location, and archival format options. Done requires an agreed design and implementation scope rather than an open-ended set of archival ideas.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.