apache / apache/druid

Even though the index_parallel task is marked failed, many of its single_phase_sub_tasks are still running.

Open
#11,396 5 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Tested on 0.21.1

### Description

Please include as much detailed information about the problem as possible.
- 15 middle managers with 20 workers each.
- We are deploying Druid inside Kubernetes.
- Each middle manager pods has 32GB RAM and 20 CPU.
- The configuration of the cluster is pretty basic, we don't use any affinity stuff.
- The native ingestion job uses maxNumConcurrentSubTasks=100. The input data is around 3TB per day with hundreds of parquet files.
- To reproduce, we just keep it running for almost a day, we will see that index_parallel is marked failed but many of the subtasks are still running.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the native ingestion index_parallel task and its single_phase_sub_tasks under the reported Kubernetes configuration. Reproduce the failure with maxNumConcurrentSubTasks=100 and inspect task lifecycle handling; done means a failed index_parallel task no longer leaves its subtasks running.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.