Azure / Azure/azure-service-bus-java

Too many ReactorThread instances in the thread dump.

Open
#365 8 comments 0 reactions 1 assignee Claimed by @yvgopal View on GitHub
question
Dominant language
Java
Stars
62
Forks
63
PR merge metrics
No merged PRs in 30d

Description

We have two subscription clients receiving messages with an interval of 30 seconds. At times, we stop receiving messages for some reason, and the we have to stop both subscription clients and reconnect them to continue receiving messages. But we can see that with each successive stop and reconnect, the total thread count of the application increases and so is the CPU usage. There are no exceptions thrown by the API during this time period. We are using servicebus version 1.2.13.

On generating a threaddump we could see that out of the 594 threads running on the application at that instant, 524 belonged to instances of ReactorThread. An example stack trace is as follows.
`"ReactorThread645d2ac6-eaeb-4599-aa24-7cff615082dc" #33560 prio=5 os_prio=64 tid=0x00000001023bc000 nid=0x832c runnable [0xffffffff118fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007aad4da68> (a sun.nio.ch.Util$3)
- locked <0x00000007aad4da58> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007aad4da78> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.qpid.proton.reactor.impl.SelectorImpl.select(SelectorImpl.java:126)
at org.apache.qpid.proton.reactor.impl.IOHandler.handleQuiesced(IOHandler.java:57)
at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:390)
at org.apache.qpid.proton.engine.BaseHandler.onReactorQuiesced(BaseHandler.java:87)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:206)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:292)
at com.microsoft.azure.servicebus.primitives.MessagingFactory$RunReactor.run(MessagingFactory.java:483)
at java.lang.Thread.run(Thread.java:745)`

All of these instances are also in thread state runnable.

It seems that subscriptionClient.close() doesn't actually close the Reactor Thread instance. We checked the heap space utilization and it follows a normal sawtooth pattern without ever crossing 200MB, so no memory leaks.

Could you please confirm if subscriptionClient.close() would actually close the reactor instance too? If not, how to do it. This high CPU usage has been a priority issue for quite some time for our clients. So a quick reply is much appreciated

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.