Even though the index_parallel task is marked failed, many of its single_phase_sub_tasks are still running.
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Tested on 0.21.1
### Description
Please include as much detailed information about the problem as possible.
- 15 middle managers with 20 workers each.
- We are deploying Druid inside Kubernetes.
- Each middle manager pods has 32GB RAM and 20 CPU.
- The configuration of the cluster is pretty basic, we don't use any affinity stuff.
- The native ingestion job uses maxNumConcurrentSubTasks=100. The input data is around 3TB per day with hundreds of parquet files.
- To reproduce, we just keep it running for almost a day, we will see that index_parallel is marked failed but many of the subtasks are still running.
Contributor guide
Research direction
Start by tracing the native ingestion index_parallel task and its single_phase_sub_tasks under the reported Kubernetes configuration. Reproduce the failure with maxNumConcurrentSubTasks=100 and inspect task lifecycle handling; done means a failed index_parallel task no longer leaves its subtasks running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100