A sudden increase of epollWait() CPU utilization after a particular commit.
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
Greetings! I am a master's student conducting research on performance diagnosis. My current focus is on studying the evolution of distributed systems and the occurrence of performance regression during their development. As part of my research, I am examining Apache Ignite as a case study.
To conduct my study, I used the YCSB benchmark (https://github.com/brianfrankcooper/YCSB/tree/master/ignite), configuring it to perform 1,000,000 update operations and initialize with 100,000 records. I conducted the testing with a single thread and 3 nodes on a single machine using 3 different ports. Upon analyzing the results, I observed that the average update latency increased by approximately 10% from version 2.7.6 to version 2.14.0 (compiled by jdk8 locally). I am currently attempting to understand the cause behind this increase. For your reference, I have included the detailed configuration of one of the nodes below.
```
10.1.0.16:47500
10.1.0.16:47501
10.1.0.16:47502
```
Next, I utilized JFR to profile the runtime and discovered a significant increase in the profiling samples of sun.nio.ch.EPollArrayWrapper.epollWait(). Specifically, the overall method profiling samples increased by approximately 2000 (equivalent to around 10% of v2.7.6's overall method profiling samples, which matches the increase ratio of latency), while the increase in samples of epollWait() was around 5500, and java.net.PlainSocketImpl.socketAccept() decreased by approximately 3900 samples. To gather this data, I conducted ten rounds of testing using the benchmark mentioned above on an Ubuntu 18.04.1 LTS system with x86_64 arch, 16 cores, and 132GB memory.
Based on my findings, I concluded that epollWait() is the primary cause of the latency increase. I have since attempted to narrow down the issue and locate the particular commit(s) responsible. I discovered that commit [1094fff](https://github.com/apache/ignite/commit/1094fff4df636fec1c807a8572597c5178a34b32) had an increase in samples of approximately 3400 compared to its [parent](https://github.com/apache/ignite/commit/5e37db2f27a02b5c31c8b7a76b18f42e9d243111). This increase accounts for around 61.82% of the total increase in epollWait() samples from v2.7.6 to v2.14.0. However, the latency did not significantly increase from its parent, as socketAccept() also decreased by approximately 3400 samples.
At this point, I would like to understand why this particular commit caused such a significant increase in epollWait() samples. The commit appears to only disable JMX monitoring, but since I lack context, I am seeking suggestions on where to focus my further study. Can you please provide any advice or recommendations?
Contributor guide
Research direction
Reproduce the YCSB workload on Apache Ignite 2.7.6 and 2.14.0 using the supplied configuration, then compare JFR profiles around commit 1094fff and its parent 5e37db2. Investigate why epollWait() samples rise while socketAccept() samples fall, and document whether the change explains the reported latency regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100