ChainSafe / ChainSafe/lodestar

Surface remaining undashboarded BLS thread pool metrics

Open
#9,408 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

## Context

While adding the same-message retry series in #, an audit of `lodestar_bls*` metrics defined in `packages/beacon-node/src/metrics/metrics/lodestar.ts` against every panel in `dashboards/*.json` turned up several metrics that are emitted in production but never plotted. The same-message retries were addressed in that PR; the rest are tracked here.

## Metrics with no dashboard coverage

| Metric | Defined at | Incremented at |
|---|---|---|
| `lodestar_bls_thread_pool_batchable_sig_sets_total` | `lodestar.ts:500` | `multithread/index.ts:168` |
| `lodestar_bls_thread_pool_prioritized_sig_sets_total` | `lodestar.ts:496` | `multithread/index.ts:165` |
| `lodestar_bls_thread_pool_sig_sets_total` | `lodestar.ts:492` | — |
| `lodestar_bls_thread_pool_error_aggregate_signature_sets_count` | `lodestar.ts:415` | — |
| `lodestar_bls_worker_thread_time_per_sigset_seconds` | `lodestar.ts:487` | per-sigset histogram |
| `lodestar_bls_single_thread_time_seconds` | `lodestar.ts:519` | single-thread fallback path |
| `lodestar_bls_single_thread_time_per_sigset_seconds` | `lodestar.ts:524` | single-thread fallback path |

Line numbers are point-in-time references — verify against current `unstable` before editing.

## Proposed placement (`dashboards/lodestar_bls_thread_pool.json`)

- **"BLS thread pool - sig sets / sec" or "% of sigs verified in batch"** — natural home for `batchable_sig_sets_total`, `prioritized_sig_sets_total`, and `sig_sets_total` as supporting series showing the composition of incoming sets and which optimization path each took.
- **"BLS thread pool - Error rates"** — add `error_aggregate_signature_sets_count` next to the existing `batch_retries` and `same_message_*` series; it's another failure counter and sits alongside them in code.
- **"BLS worker pool" row / "sig set verification time / set"** — add `bls_worker_thread_time_per_sigset_seconds`, `bls_single_thread_time_seconds`, and `bls_single_thread_time_per_sigset_seconds` so the single-thread fallback path is observable alongside the thread-pool path.

## Acceptance

- [ ] Each metric above is queried by at least one panel in `dashboards/`.
- [ ] `scripts/validate-grafana-dashboards.sh` passes.
- [ ] Each new panel/series has a description explaining what "healthy" looks like (since most of these are diagnostic counters whose meaning is non-obvious to oncall).

## AI disclosure

Issue drafted with assistance from Claude (Opus 4.7).

Contributor guide

Open the contributing guide

Research direction

Compare the listed metrics in packages/beacon-node/src/metrics/metrics/lodestar.ts with dashboards/lodestar_bls_thread_pool.json and the other dashboards, verifying locations against current unstable. Add coverage for each metric in the proposed panels with descriptions of healthy behavior, then run scripts/validate-grafana-dashboards.sh and confirm every metric is queried.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, typescript
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.