PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
Mailing list discussion: https://lists.apache.org/thread/pkkz2jgwtzpksp6d4rdm1pyxzb3z6vmg
## Motivation
Since Pulsar Admin API uses the blocking servlet API, all Jetty threads might be occupied and this causes unavailability of the Pulsar Admin API. The default value for the maximum number of threads for Jetty is too low in Pulsar. That is the root cause of many problems where Pulsar Admin API is unavailable when all threads are in use.
## Additional context
- Examples of previous issues where Jetty threads have been occupied and caused problems: #13666 #4756 #10619
- Mailing list thread about "make async" changes: https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l
- Additional changes to Jetty configuration: #14353
## Implementation
- Jetty defaults to 200 maximum threads, to prevent thread pool starvation. Make Pulsar use the same default value by setting `numHttpServerThreads=200`.
- Update the documentation for `numHttpServerThreads`
- The PR is already in place: https://github.com/apache/pulsar/pull/14320
- Set Jetty selectors and acceptors parameters to `-1` so that Jetty automatically chooses optimal values based on available cores. The rationale is explained in the Q&A below.
- A separate PR will be made for this change.
## Q&A
### Q: What's the reason of setting the default value to 200? If the node just have one core, what will happen?
These are threads. Jetty defaults to 200 maximum threads, to prevent thread pool starvation. This is recommended when using blocking Servlet API. The problem is that Pulsar uses the blocking servlet API and doesn't have a sufficient number of threads which are needed and recommended.
The value 200 doesn't mean that there will be 200 threads to start with. This is the maximum size for the thread pool. When the value is more than 8, Jetty will start with 8 initial threads and add more threads to the pool when all threads are occupied.
### Q: Do we need to take the number of system cores into consideration for the maximum threads of the thread pool?
No. Jetty is different from Netty in this aspect. In Netty, everything should be asynchronous and "thou shall never block". In Jetty, the maximum number of threads for the thread pool should be set to 50-500 threads and blocking operations are fine.
The recommendation for the thread pool is explained in Jetty documentation https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_thread_pool
> Thread Pool
> Configure with goal of limiting memory usage maximum available. Typically this is >50 and <500
However, there are separate settings which should take the number of available processors (cores) into account in Jetty.
http port acceptor and selector count:
https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L88
https port acceptor and selector count:
https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L125
Jetty [documentantion for acceptors](https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_acceptors):
> Acceptors
> The standard rule of thumb for the number of Accepters to configure is one per CPU on a given machine.
Jetty [documentation for selectors](https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/ServerConnector.html):
> Selectors
> The default number of selectors is equal to half of the number of processors available to the JVM, which should allow optimal performance even if all the connections used are performing significant non-blocking work in the callback tasks.
The settings in jetty are the "acceptor" and "selector" thread count settings. These have been fixed to 1 in Pulsar.
The `acceptors` and `selectors` settings should be both set to -1. Jetty would pick the recommended count based on cores in that case.
Contributor guide
Research direction
Start by reading the Jetty configuration in pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java, including the HTTP and HTTPS acceptor and selector settings referenced in the issue. Review the existing PR for the numHttpServerThreads documentation and confirm that the proposed defaults and documentation changes are covered; the separate acceptor and selector change should be treated independently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100