ChainSafe / ChainSafe/lodestar

Network thread performance issue due to async randomness

Open
#7,801 0 comments 0 reactions 0 assignees View on GitHub
meta-bug scope-networking scope-performance
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

This is a review of metrics monitored on our test mainnet node of #7761, it's very likely we'll merge that PR since the issue only happens on a test mainnet node subscribing on all subnets and it improved the mainnet thread a lot, so I make this issue for later reference

- in general, that PR improves the main thread a lot that cause more pressure on the network thread

Image

- on the last 8 days, scavenge gc keeps going up

Image

- the event loop lag keeps increasing

Image

- due to that the request I/O time increased, especially for ping, status, metadata

Image

- the node has so many peers so it has to disconnect a lot of them

Image

Image

- peer manager heart beat also increased

Image

- on the main thread, it improved a lot

Image

Image

The issue does not happen on other nodes

### Expected behavior

Event loop lag on the network thread is the same to before

### Steps to reproduce

_No response_

### Additional context

_No response_

### Operating system

Linux

### Lodestar version or commit hash

mkeil/aggregate-with-randomness-async-again

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.