ChainSafe / ChainSafe/lodestar
Surface remaining undashboarded BLS thread pool metrics
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
## Context
While adding the same-message retry series in #, an audit of `lodestar_bls*` metrics defined in `packages/beacon-node/src/metrics/metrics/lodestar.ts` against every panel in `dashboards/*.json` turned up several metrics that are emitted in production but never plotted. The same-message retries were addressed in that PR; the rest are tracked here.
## Metrics with no dashboard coverage
| Metric | Defined at | Incremented at |
|---|---|---|
| `lodestar_bls_thread_pool_batchable_sig_sets_total` | `lodestar.ts:500` | `multithread/index.ts:168` |
| `lodestar_bls_thread_pool_prioritized_sig_sets_total` | `lodestar.ts:496` | `multithread/index.ts:165` |
| `lodestar_bls_thread_pool_sig_sets_total` | `lodestar.ts:492` | — |
| `lodestar_bls_thread_pool_error_aggregate_signature_sets_count` | `lodestar.ts:415` | — |
| `lodestar_bls_worker_thread_time_per_sigset_seconds` | `lodestar.ts:487` | per-sigset histogram |
| `lodestar_bls_single_thread_time_seconds` | `lodestar.ts:519` | single-thread fallback path |
| `lodestar_bls_single_thread_time_per_sigset_seconds` | `lodestar.ts:524` | single-thread fallback path |
Line numbers are point-in-time references — verify against current `unstable` before editing.
## Proposed placement (`dashboards/lodestar_bls_thread_pool.json`)
- **"BLS thread pool - sig sets / sec" or "% of sigs verified in batch"** — natural home for `batchable_sig_sets_total`, `prioritized_sig_sets_total`, and `sig_sets_total` as supporting series showing the composition of incoming sets and which optimization path each took.
- **"BLS thread pool - Error rates"** — add `error_aggregate_signature_sets_count` next to the existing `batch_retries` and `same_message_*` series; it's another failure counter and sits alongside them in code.
- **"BLS worker pool" row / "sig set verification time / set"** — add `bls_worker_thread_time_per_sigset_seconds`, `bls_single_thread_time_seconds`, and `bls_single_thread_time_per_sigset_seconds` so the single-thread fallback path is observable alongside the thread-pool path.
## Acceptance
- [ ] Each metric above is queried by at least one panel in `dashboards/`.
- [ ] `scripts/validate-grafana-dashboards.sh` passes.
- [ ] Each new panel/series has a description explaining what "healthy" looks like (since most of these are diagnostic counters whose meaning is non-obvious to oncall).
## AI disclosure
Issue drafted with assistance from Claude (Opus 4.7).
Contributor guide
Research direction
Compare the listed metrics in packages/beacon-node/src/metrics/metrics/lodestar.ts with dashboards/lodestar_bls_thread_pool.json and the other dashboards, verifying locations against current unstable. Add coverage for each metric in the proposed panels with descriptions of healthy behavior, then run scripts/validate-grafana-dashboards.sh and confirm every metric is queried.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, typescript
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100