apache / apache/druid

Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?

Open
#12,898 7 comments 1 reaction 0 assignees View on GitHub
Area - Segment Balancing/Coordination
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

**Apache Druid Version:** 0.23.0

**Problem**:
Sometimes when we have infrastructure issues, we have to decommission some Historicals and temporarily lower retention period (to not run out of disk on running ones). When we bring back the Historicals and re-apply our old retention period and reload the dropped data, our ingestion tasks wait for handoff much longer and sometimes fail because they did not handoff completely in the completionTimeout phase even if that is set to a higher number.

Is it possible to adjust Coordinator behavior to prioritize new segment handoffs from currently running ingestion tasks over reloading dropped data so that running ingestion tasks are not affected and still have some segments being reloaded?

Contributor guide

Open the contributing guide

Research direction

Start by tracing Apache Druid's Coordinator segment handoff and ingestion task completionTimeout behavior. Compare handoffs for new segments from running ingestion tasks with reloads of dropped data, then define an implementation path and tests showing that running-task handoffs are prioritized without preventing reloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.