Concurrency issue on high core count systems
- Dominant language
- Java
- Stars
- 593
- Forks
- 123
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 4
Description
Request processing gets stuck on high core count systems.
Not yet sure what causes this issue but I was able to reproduce it with the demo image of 1.2.1 and 1.4.0 using:
```ab -n 20000 -c 2000 http://server:8080/api/v1/demo/webroot/automobiles```
```
21:48:26.336 [] ERROR [vert.x-eventloop-thread-6] [c.g.m.r.r.FailureHandler] - Error in request for path {GET /api/v1/demo/webroot/automobiles}
java.lang.RuntimeException: java.util.concurrent.TimeoutException: The source did not signal an event for 40 seconds and has been terminated.
at io.reactivex.internal.util.ExceptionHelper.wrapOrThrow(ExceptionHelper.java:46)
at io.reactivex.internal.observers.BlockingMultiObserver.blockingGet(BlockingMultiObserver.java:93)
at io.reactivex.Single.blockingGet(Single.java:2737)
at com.gentics.mesh.graphdb.spi.Database.lambda$null$3(Database.java:134)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException: The source did not signal an event for 40 seconds and has been terminated.
at io.reactivex.internal.operators.single.SingleTimeout$TimeoutMainObserver.run(SingleTimeout.java:123)
at io.reactivex.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:38)
at io.reactivex.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:26)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
... 1 common frames omitted
```
* `-c 16` - works
* `-c 32` - works
* `-c 40` - fails
* `-c 64` - fails
CPU: Xeon E2699 v3 (18 Core - 32 HT)
Env: Docker Image
Contributor guide
Research direction
Start by reproducing the issue with the demo image and the provided ApacheBench command, comparing concurrency levels from 16 through 64. Read the stack trace around Database.java:134 and investigate why requests time out above the reported threshold. Done means high-concurrency requests no longer become stuck or hit the 40-second timeout under the reproduction scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100