Orphan segments in ceph
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
The Druid version 33.0.0.
### Description
We have identified a potential issue with orphaned segments in our deployment, which utilizes Ceph as deep storage and Postgres as the metadata store.
Several anomalies have been observed:
- A significant number of segments are present in Ceph but missing from Postgres metadata.
- Some of these segments are very old, have exceeded their retention period, and were never cleaned up.
- A subset of segments had never been loaded by the cluster because they did not exist in Postgres at all, implying they were unknown to the coordinator.
- After manually deleting these segments from Ceph, there were no related errors or recovery attempts from the cluster, and Ceph disk usage dropped noticeably, confirming they were unused and orphaned.
- the steps we took for removing were:
- list segments from Ceph
- list from postgres using payload field from druid_segments table ([payload] [loadSpec] [key])
- check differences and remove keys that were not in PostgreSQL and existed on Ceph storage
Additional context:
- These segments appear to be completely unmanaged by Druid since their metadata entries never existed or were removed prematurely.
- Manual deletion did not cause any segment load/unload events, coordinator log warnings, or missing segment alerts.
Contributor guide
Research direction
Begin with Druid 33.0.0's segment metadata and deep-storage lifecycle, comparing Ceph object keys with the druid_segments.payload.loadSpec.key values in Postgres. Use the reported difference-check and manual-removal procedure as the starting investigation; done requires explaining how the metadata and Ceph contents diverged and documenting a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgres
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100