VirtualStorage implementation for MSQ in druid 37 causing threadpool leak causing native threads exhausted
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
We have using druid 37 and we are facing issue with threadpool not closing for MSQ as its not releasing the threads on task exit. we are exhausting the native threads due to this threads leak. once the threads hit the ulimit value it doesnt allow any new threads creation until we restart the indexer service. The issue is more visible in indexer as MSQ runs on same jvm unlike middle manager where each MSQ will fork different JVM.
```
[root@xxxxx ~]# ls /proc/32589/task | wc -l
254,869
[root@xxxxx ~]#
```
254,869 ← the leaker, ~99% thread are VirtualStorageOnDemandLoadingThread
Looks the issue was identified and fixed : dedicated ephemeral storage loading pool for tasks- #19658
#19658
I could see the backporting is done only to druid 38. Can we backport to druid 37 as its is a major break on druid stability running MSQ running on indexers ?
Contributor guide
Research direction
Start by comparing the Druid 37 MSQ VirtualStorage implementation with the dedicated ephemeral storage loading pool fix in issue #19658 and its Druid 38 backport. Verify that VirtualStorageOnDemandLoadingThread resources are released when MSQ tasks exit on indexers, and add or run regression coverage showing that repeated task exits do not leak threads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100