apache / apache/pinot

Pinot returns out of retention data until the entire segment is deleted.

Open
#16,689 4 comments 0 reactions 2 assignees Claimed by @pri1712 View on GitHub
PEP-Request
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

### Problem

Pinot currently uses TimeRetentionStrategy to remove segments based on their end time (the maximum time value in the segment). This means a segment remains queryable as long as its end time is within retention. If a segment spans multiple days (e.g., day x and x+1), it stays available until x+1 falls out of retention, even though records from x are already expired.

This issue becomes more prominent when small segment merger is enabled, since compacted segments often contain multiple days of data. As a result, queries may return out-of-retention records.

### Proposal

Introduce an option to automatically exclude out-of-retention records at query time. Pinot could apply a time filter based on the configured retention boundary, ensuring only valid data is returned.

This behavior should initially be opt-in, controlled through a query option. For example:

```
SET skipOutOfRetentionValues=true;
SELECT COUNT(*) FROM myTable;

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.