Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
**Apache Druid Version:** 0.23.0
**Problem**:
Sometimes when we have infrastructure issues, we have to decommission some Historicals and temporarily lower retention period (to not run out of disk on running ones). When we bring back the Historicals and re-apply our old retention period and reload the dropped data, our ingestion tasks wait for handoff much longer and sometimes fail because they did not handoff completely in the completionTimeout phase even if that is set to a higher number.
Is it possible to adjust Coordinator behavior to prioritize new segment handoffs from currently running ingestion tasks over reloading dropped data so that running ingestion tasks are not affected and still have some segments being reloaded?
Contributor guide
Research direction
Start by tracing Apache Druid's Coordinator segment handoff and ingestion task completionTimeout behavior. Compare handoffs for new segments from running ingestion tasks with reloads of dropped data, then define an implementation path and tests showing that running-task handoffs are prioritized without preventing reloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100