apache / apache/druid

VirtualStorage implementation for MSQ in druid 37 causing threadpool leak causing native threads exhausted

Open
#19,904 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

We have using druid 37 and we are facing issue with threadpool not closing for MSQ as its not releasing the threads on task exit. we are exhausting the native threads due to this threads leak. once the threads hit the ulimit value it doesnt allow any new threads creation until we restart the indexer service. The issue is more visible in indexer as MSQ runs on same jvm unlike middle manager where each MSQ will fork different JVM.

```
[root@xxxxx ~]# ls /proc/32589/task | wc -l
254,869
[root@xxxxx ~]#
```
254,869 ← the leaker, ~99% thread are VirtualStorageOnDemandLoadingThread

Looks the issue was identified and fixed : dedicated ephemeral storage loading pool for tasks- #19658
#19658
I could see the backporting is done only to druid 38. Can we backport to druid 37 as its is a major break on druid stability running MSQ running on indexers ?

Contributor guide

Open the contributing guide

Research direction

Start by comparing the Druid 37 MSQ VirtualStorage implementation with the dedicated ephemeral storage loading pool fix in issue #19658 and its Druid 38 backport. Verify that VirtualStorageOnDemandLoadingThread resources are released when MSQ tasks exit on indexers, and add or run regression coverage showing that repeated task exits do not leak threads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.