apache / apache/druid

Orphan segments in ceph

Open
#18,788 1 comment 0 reactions 0 assignees View on GitHub
Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

The Druid version 33.0.0.

### Description

We have identified a potential issue with orphaned segments in our deployment, which utilizes Ceph as deep storage and Postgres as the metadata store.

Several anomalies have been observed:

- A significant number of segments are present in Ceph but missing from Postgres metadata.
- Some of these segments are very old, have exceeded their retention period, and were never cleaned up.
- A subset of segments had never been loaded by the cluster because they did not exist in Postgres at all, implying they were unknown to the coordinator.
- After manually deleting these segments from Ceph, there were no related errors or recovery attempts from the cluster, and Ceph disk usage dropped noticeably, confirming they were unused and orphaned.
- the steps we took for removing were:
- list segments from Ceph
- list from postgres using payload field from druid_segments table ([payload] [loadSpec] [key])
- check differences and remove keys that were not in PostgreSQL and existed on Ceph storage

Additional context:

- These segments appear to be completely unmanaged by Druid since their metadata entries never existed or were removed prematurely.
- Manual deletion did not cause any segment load/unload events, coordinator log warnings, or missing segment alerts.

Contributor guide

Open the contributing guide

Research direction

Begin with Druid 33.0.0's segment metadata and deep-storage lifecycle, comparing Ceph object keys with the druid_segments.payload.loadSpec.key values in Postgres. Use the reported difference-check and manual-removal procedure as the starting investigation; done requires explaining how the metadata and Ceph contents diverged and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgres
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.