apache / apache/pinot

A segment file is not removed (moved to Deleted_Segments folder) from an S3 backed segment store when a segment is deleted

Open
#8,116 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Steps to reproduce
1. Setup controller with an S3 backed segment store
2. Create a segment using LaunchDataIngestionJob with a job type of `SegmentCreationAndMetadataPush`

The segment metadata will look something like the following:
```
{
"id": "actions",
"simpleFields": {
"segment.crc": "2803120123",
"segment.creation.time": "1643896475639",
"segment.index.version": "v3",
"segment.name": "actions",
"segment.offline.download.url": "s3:///bi-analytics/pinot/deepstorage/actions/actions.tar.gz",
"segment.offline.push.time": "1643896476013",
"segment.table.name": "actions",
"segment.total.docs": "12",
"segment.type": "OFFLINE"
},
"mapFields": {
"custom.map": {
"input.data.file.uri": "file://bi-analytics/input/actions/2021/01/01/actions.json"
}
},
"listFields": {}
}
```

From the UI, navigate to the newly created segment and delete it. The expected outcome is the segment file `s3:///bi-analytics/pinot/deepstorage/actions/actions.tar.gz` is removed (moved to Deleted_Segments folder) from the segment store and the segment metadata is removed from Zookeeper.

The actual outcome is the segment metadata is removed from Zookeeper and the SegmentDeletionManager ["Fails to find local segment file for segment"](https://github.com/apache/pinot/blob/4561b343529da8433e8ff132e0cc2a324acfda56/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L212). This is because the `fileToDeleteURI`[variable](https://github.com/apache/pinot/blob/4561b343529da8433e8ff132e0cc2a324acfda56/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L181) contains `s3:///bi-analytics/pinot/deepstorage/actions`, which S3PinotFS.exists(...) returns as false.

See screen shot of debug session for further details:

![debug-session](https://user-images.githubusercontent.com/18295326/152370631-ad300409-7202-4487-8077-3841e66a4603.png)

The issue also manifests itself when data retention on a table is configured, that is, the table's segment(s) are not removed from the S3 segment store after the retention period has expired.

Segment deletion works fine when the segment store is backed by a regular file system.

Contributor guide

Open the contributing guide

Research direction

Start in pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java, especially the fileToDeleteURI handling and the reported exists check. Compare that path with S3PinotFS.exists(...) and reproduce deletion through the UI or retention flow. Done means the S3 segment is moved to Deleted_Segments and its ZooKeeper metadata is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.