grpc / grpc/grpc-java

Excess contention in ExecutorService

Open
#2,118 8 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

When profiling a client with 200K active RPCs, there is a point of contention on the Executor. Each RPC gets its own SerializingExecutor, which executes work on an underlying executor. Currently, that executor is ThreadPoolExecutor in almost all cases, which itself has a BlockingQueue. That queue is heavily contended showing up at _minutes_ of wasted time:

```
141.17mins 79.41% 79.41% 141.22mins 79.44% java.util.concurrent.LinkedBlockingQueue.offer LinkedBlockingQueue.java
36.51mins 20.54% 99.95% 36.52mins 20.54% java.util.concurrent.LinkedBlockingQueue.take LinkedBlockingQueue.java
```

An idea to fix this is to have some sort of striping executor in order to prevent this contention from happening.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.