Number of "pulsar-timer-*" threads increases sporadically over time
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Describe the bug**
The number of "pulsar-timer-*" threads increases sporadically over time and ends up consuming many CPU cycles. Each thread uses a few percent of CPU, but the number of threads created ends up with a lot of CPU used despite the system being idle. On startup, we will observe roughly 10 of these threads but when we check again in 12-48 hours, we will often see several pages of these threads.
**To Reproduce**
At this point, I have not seen the trigger that leads to the increase in the number of threads, but the issue might be due to network instability.
Though I am not sure what caused it, this exception was observed recently when the number of threads increased:
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at org.apache.pulsar.common.util.collections.GrowableArrayBlockingQueue.poll(GrowableArrayBlockingQueue.java:203)
at org.apache.pulsar.client.impl.ConsumerImpl.internalReceive(ConsumerImpl.java:337)
**Expected behavior**
If the number of "pulsar-timer-*" threads increases due to load, it should also decrease when demand is reduced.
**Screenshots**
Top output showing threads looks as follows with pages of pulsar-time-* threads
15989 root 20 0 3732756 762780 27260 R 2.0 9.3 8:09.58 pulsar-time+
15993 root 20 0 3732756 762780 27260 S 2.0 9.3 8:09.22 pulsar-time+
15994 root 20 0 3732756 762780 27260 S 2.0 9.3 8:09.22 pulsar-time+
15914 root 20 0 3732756 762780 27260 S 2.0 9.3 8:06.07 C1 Compiler+
15990 root 20 0 3732756 762780 27260 S 2.0 9.3 8:09.49 pulsar-time+
15991 root 20 0 3732756 762780 27260 R 2.0 9.3 8:09.65 pulsar-time+
15992 root 20 0 3732756 762780 27260 R 2.0 9.3 8:09.57 pulsar-time+
15996 root 20 0 3732756 762780 27260 R 2.0 9.3 8:08.29 pulsar-time+
15998 root 20 0 3732756 762780 27260 R 2.0 9.3 8:08.09 notificatio+
16014 root 20 0 3732756 762780 27260 R 2.0 9.3 8:09.36 pulsar-time+
16015 root 20 0 3732756 762780 27260 R 1.0 9.3 8:09.13 pulsar-time+
16020 root 20 0 3732756 762780 27260 S 1.0 9.3 8:00.63 pulsar-clie+
16022 root 20 0 3732756 762780 27260 R 1.0 9.3 8:09.40 pulsar-time+
16035 root 20 0 3732756 762780 27260 R 1.0 9.3 8:09.04 pulsar-time+
...
**Desktop (please complete the following information):**
- Running on patch current Ubuntu 18.* release.
- running the 2.4.2 version of pulsar
**Additional context**
NA
Contributor guide
Research direction
Start by reproducing the thread growth on Apache Pulsar 2.4.2 with the reported Ubuntu 18.* environment, while monitoring pulsar-timer-* threads over 12–48 hours. Trace the reported InterruptedException through GrowableArrayBlockingQueue.poll and ConsumerImpl.internalReceive, then identify the trigger and verify that thread counts decrease when demand is reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100