The coordinator doesn't clean up the pendingSegments table for a datasource if there is no published segments in the datasource
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
All versions since 0.12.0
### Description
`DruidCoordinatorCleanupPendingSegments` checks only the datasources that have at least one published segments. See the below code snippet.
```java
for (String dataSource : params.getUsedSegmentsTimelinesPerDataSource().keySet()) {
if (!params.getCoordinatorDynamicConfig().getDataSourcesToNotKillStalePendingSegmentsIn().contains(dataSource)) {
log.info(
"Killed [%d] pendingSegments created until [%s] for dataSource[%s]",
indexingServiceClient.killPendingSegments(dataSource, stalePendingSegmentsCutoffCreationTime),
stalePendingSegmentsCutoffCreationTime,
dataSource
);
}
}
```
An easy fix would be adding a new class that handles the access to the `pendingSegments` table.
Contributor guide
Research direction
Start with DruidCoordinatorCleanupPendingSegments and the shown loop over getUsedSegmentsTimelinesPerDataSource(), then trace how pendingSegments is accessed for datasources with no published segments. Verify the behavior for an empty datasource and define completion as stale pending segments being cleaned up there without affecting the exclusion configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100