Azure / Azure/azure-service-bus-java
Too many ReactorThread instances in the thread dump.
- Dominant language
- Java
- Stars
- 62
- Forks
- 63
- PR merge metrics
- No merged PRs in 30d
Description
We have two subscription clients receiving messages with an interval of 30 seconds. At times, we stop receiving messages for some reason, and the we have to stop both subscription clients and reconnect them to continue receiving messages. But we can see that with each successive stop and reconnect, the total thread count of the application increases and so is the CPU usage. There are no exceptions thrown by the API during this time period. We are using servicebus version 1.2.13.
On generating a threaddump we could see that out of the 594 threads running on the application at that instant, 524 belonged to instances of ReactorThread. An example stack trace is as follows.
`"ReactorThread645d2ac6-eaeb-4599-aa24-7cff615082dc" #33560 prio=5 os_prio=64 tid=0x00000001023bc000 nid=0x832c runnable [0xffffffff118fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007aad4da68> (a sun.nio.ch.Util$3)
- locked <0x00000007aad4da58> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007aad4da78> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.qpid.proton.reactor.impl.SelectorImpl.select(SelectorImpl.java:126)
at org.apache.qpid.proton.reactor.impl.IOHandler.handleQuiesced(IOHandler.java:57)
at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:390)
at org.apache.qpid.proton.engine.BaseHandler.onReactorQuiesced(BaseHandler.java:87)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:206)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:292)
at com.microsoft.azure.servicebus.primitives.MessagingFactory$RunReactor.run(MessagingFactory.java:483)
at java.lang.Thread.run(Thread.java:745)`
All of these instances are also in thread state runnable.
It seems that subscriptionClient.close() doesn't actually close the Reactor Thread instance. We checked the heap space utilization and it follows a normal sawtooth pattern without ever crossing 200MB, so no memory leaks.
Could you please confirm if subscriptionClient.close() would actually close the reactor instance too? If not, how to do it. This high CPU usage has been a priority issue for quite some time for our clients. So a quick reply is much appreciated
Contributor guide
Assessment
This issue has not been assessed yet.