gentics / gentics/mesh

Concurrency issue on high core count systems

Open
#1,006 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
593
Forks
123
Avg merge
2d 18h
Merged PRs (30d)
4

Description

Request processing gets stuck on high core count systems.

Not yet sure what causes this issue but I was able to reproduce it with the demo image of 1.2.1 and 1.4.0 using:

```ab -n 20000 -c 2000 http://server:8080/api/v1/demo/webroot/automobiles```

```
21:48:26.336 [] ERROR [vert.x-eventloop-thread-6] [c.g.m.r.r.FailureHandler] - Error in request for path {GET /api/v1/demo/webroot/automobiles}
java.lang.RuntimeException: java.util.concurrent.TimeoutException: The source did not signal an event for 40 seconds and has been terminated.
at io.reactivex.internal.util.ExceptionHelper.wrapOrThrow(ExceptionHelper.java:46)
at io.reactivex.internal.observers.BlockingMultiObserver.blockingGet(BlockingMultiObserver.java:93)
at io.reactivex.Single.blockingGet(Single.java:2737)
at com.gentics.mesh.graphdb.spi.Database.lambda$null$3(Database.java:134)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException: The source did not signal an event for 40 seconds and has been terminated.
at io.reactivex.internal.operators.single.SingleTimeout$TimeoutMainObserver.run(SingleTimeout.java:123)
at io.reactivex.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:38)
at io.reactivex.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:26)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
... 1 common frames omitted
```

* `-c 16` - works
* `-c 32` - works
* `-c 40` - fails
* `-c 64` - fails

CPU: Xeon E2699 v3 (18 Core - 32 HT)
Env: Docker Image

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the demo image and the provided ApacheBench command, comparing concurrency levels from 16 through 64. Read the stack trace around Database.java:134 and investigate why requests time out above the reported threshold. Done means high-concurrency requests no longer become stuck or hit the 40-second timeout under the reproduction scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.