apache / apache/druid

Auto compaction doesn't work when the overlord runs in the "local" mode

Open
#8,683 0 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

All versions

### Description

In auto compaction, the coordinator gets the worker (middleManager/indexer) information from the overlord to compute the total number of task slots. But if the overlord runs in the local mode, it will return a 500 error with "Task Runner does not support worker listing" which in turn leading to the failure of auto compaction.

I think we can just simply use only `maxCompactionTaskSlots` if the overlord is running in the local mode. Slightly related, the default value of `maxCompactionTaskSlots` is `Integer.MAX_VALUE` and could be limited to some value.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing auto compaction's coordinator lookup of worker information through the overlord and its local-mode task runner. Done means local mode no longer fails on unsupported worker listing and uses maxCompactionTaskSlots instead; verify the relevant auto-compaction behavior with tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.