ChainSafe / ChainSafe/lodestar

Improve worker event loop lag monitoring

Open
#6,720 7 comments 0 reactions 1 assignee Claimed by @jeluard View on GitHub
scope-performance
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

[collectNodeJSMetrics](https://github.com/ChainSafe/lodestar/blob/122a0f8822711c7b3032b3bd7c6d02e603953d93/packages/beacon-node/src/metrics/nodeJsMetrics.ts#L5) relies on [prom-client](https://github.com/siimon/prom-client/blob/e4c0a5410c9ffba89664891c97b4d47cbb6ac9f6/lib/metrics/eventLoopLag.js#L47) to collect `eventLoopMonitoring`. This in turns relies on [node:perf_hooks/monitorEventLoopDelay](https://nodejs.org/api/perf_hooks.html#perf_hooksmonitoreventloopdelayoptions) that only works for the `main` thread or `worker` from the `cluster` module (as documented [here](https://github.com/siimon/prom-client?tab=readme-ov-file#usage-with-nodejss-cluster-module)).
`woker_threads` support is unclear, although we relies on this for `worker_threads` monitoring ([network](https://github.com/ChainSafe/lodestar/blob/122a0f8822711c7b3032b3bd7c6d02e603953d93/packages/beacon-node/src/network/core/networkCoreWorker.ts#L50) and [discv5](https://github.com/ChainSafe/lodestar/blob/122a0f8822711c7b3032b3bd7c6d02e603953d93/packages/beacon-node/src/network/discv5/worker.ts#L36)).

# Proposed action items

Improve event loop monitoring, especially for `workers`. It looks preferable not to rely on `prom-client` for this specific monitoring and add support directly in `lodestar`.

* [ ] add [ELU](https://nodejs.org/api/perf_hooks.html#performanceeventlooputilizationutilization1-utilization2) monitoring for finer grain data ([monitorEventLoopDelay](https://nodejs.org/api/perf_hooks.html#perf_hooksmonitoreventloopdelayoptions) relies on histogram and averages)
* [ ] add new dashboards

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.