A segment file is not removed (moved to Deleted_Segments folder) from an S3 backed segment store when a segment is deleted
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Steps to reproduce
1. Setup controller with an S3 backed segment store
2. Create a segment using LaunchDataIngestionJob with a job type of `SegmentCreationAndMetadataPush`
The segment metadata will look something like the following:
```
{
"id": "actions",
"simpleFields": {
"segment.crc": "2803120123",
"segment.creation.time": "1643896475639",
"segment.index.version": "v3",
"segment.name": "actions",
"segment.offline.download.url": "s3:///bi-analytics/pinot/deepstorage/actions/actions.tar.gz",
"segment.offline.push.time": "1643896476013",
"segment.table.name": "actions",
"segment.total.docs": "12",
"segment.type": "OFFLINE"
},
"mapFields": {
"custom.map": {
"input.data.file.uri": "file://bi-analytics/input/actions/2021/01/01/actions.json"
}
},
"listFields": {}
}
```
From the UI, navigate to the newly created segment and delete it. The expected outcome is the segment file `s3:///bi-analytics/pinot/deepstorage/actions/actions.tar.gz` is removed (moved to Deleted_Segments folder) from the segment store and the segment metadata is removed from Zookeeper.
The actual outcome is the segment metadata is removed from Zookeeper and the SegmentDeletionManager ["Fails to find local segment file for segment"](https://github.com/apache/pinot/blob/4561b343529da8433e8ff132e0cc2a324acfda56/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L212). This is because the `fileToDeleteURI`[variable](https://github.com/apache/pinot/blob/4561b343529da8433e8ff132e0cc2a324acfda56/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L181) contains `s3:///bi-analytics/pinot/deepstorage/actions`, which S3PinotFS.exists(...) returns as false.
See screen shot of debug session for further details:

The issue also manifests itself when data retention on a table is configured, that is, the table's segment(s) are not removed from the S3 segment store after the retention period has expired.
Segment deletion works fine when the segment store is backed by a regular file system.
Contributor guide
Research direction
Start in pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java, especially the fileToDeleteURI handling and the reported exists check. Compare that path with S3PinotFS.exists(...) and reproduce deletion through the UI or retention flow. Done means the S3 segment is moved to Deleted_Segments and its ZooKeeper metadata is removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100