ChainSafe / ChainSafe/lodestar
Improve worker event loop lag monitoring
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
[collectNodeJSMetrics](https://github.com/ChainSafe/lodestar/blob/122a0f8822711c7b3032b3bd7c6d02e603953d93/packages/beacon-node/src/metrics/nodeJsMetrics.ts#L5) relies on [prom-client](https://github.com/siimon/prom-client/blob/e4c0a5410c9ffba89664891c97b4d47cbb6ac9f6/lib/metrics/eventLoopLag.js#L47) to collect `eventLoopMonitoring`. This in turns relies on [node:perf_hooks/monitorEventLoopDelay](https://nodejs.org/api/perf_hooks.html#perf_hooksmonitoreventloopdelayoptions) that only works for the `main` thread or `worker` from the `cluster` module (as documented [here](https://github.com/siimon/prom-client?tab=readme-ov-file#usage-with-nodejss-cluster-module)).
`woker_threads` support is unclear, although we relies on this for `worker_threads` monitoring ([network](https://github.com/ChainSafe/lodestar/blob/122a0f8822711c7b3032b3bd7c6d02e603953d93/packages/beacon-node/src/network/core/networkCoreWorker.ts#L50) and [discv5](https://github.com/ChainSafe/lodestar/blob/122a0f8822711c7b3032b3bd7c6d02e603953d93/packages/beacon-node/src/network/discv5/worker.ts#L36)).
# Proposed action items
Improve event loop monitoring, especially for `workers`. It looks preferable not to rely on `prom-client` for this specific monitoring and add support directly in `lodestar`.
* [ ] add [ELU](https://nodejs.org/api/perf_hooks.html#performanceeventlooputilizationutilization1-utilization2) monitoring for finer grain data ([monitorEventLoopDelay](https://nodejs.org/api/perf_hooks.html#perf_hooksmonitoreventloopdelayoptions) relies on histogram and averages)
* [ ] add new dashboards
Contributor guide
Assessment
This issue has not been assessed yet.