apache / apache/pinot

Finish the missing lifecycle pieces in SingleFileIndexDirectory

Open
#18,219 1 comment 0 reactions 0 assignees View on GitHub
cleanup enhancement index java priority: low
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

### Motivation
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java` still lists missing features near the class header:
- `newBuffer` opens a separate mapped buffer
- remove-index support is incomplete
- there is no way to abort or discard writes

The class already has mark-and-sweep comments for V3 cleanup, so the remaining lifecycle gaps are documented but unfinished.

### Scope
- Decide which missing behaviors should be supported directly versus explicitly unsupported.
- Implement or document remove-index and aborted-write handling.
- Add tests around partial writes and cleanup semantics.
- Update the class-level comments to match reality.

### Notes
Observed on `upstream/master` on April 15, 2026.

Contributor guide

Open the contributing guide

Research direction

Start with pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java, especially the class-header notes and V3 mark-and-sweep comments. Trace the newBuffer, remove-index, and write paths, then define which lifecycle behaviors are supported or explicitly unsupported. Done means the implementation or documentation matches those decisions and tests cover partial writes and cleanup semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.