apache / apache/druid

The coordinator doesn't clean up the pendingSegments table for a datasource if there is no published segments in the datasource

Open
#9,245 0 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

All versions since 0.12.0

### Description

`DruidCoordinatorCleanupPendingSegments` checks only the datasources that have at least one published segments. See the below code snippet.

```java
for (String dataSource : params.getUsedSegmentsTimelinesPerDataSource().keySet()) {
if (!params.getCoordinatorDynamicConfig().getDataSourcesToNotKillStalePendingSegmentsIn().contains(dataSource)) {
log.info(
"Killed [%d] pendingSegments created until [%s] for dataSource[%s]",
indexingServiceClient.killPendingSegments(dataSource, stalePendingSegmentsCutoffCreationTime),
stalePendingSegmentsCutoffCreationTime,
dataSource
);
}
}
```

An easy fix would be adding a new class that handles the access to the `pendingSegments` table.

Contributor guide

Open the contributing guide

Research direction

Start with DruidCoordinatorCleanupPendingSegments and the shown loop over getUsedSegmentsTimelinesPerDataSource(), then trace how pendingSegments is accessed for datasources with no published segments. Verify the behavior for an empty datasource and define completion as stale pending segments being cleaned up there without affecting the exclusion configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.