erigontech / erigontech/erigon

cl: no metric for local-vs-relay block production source

Open
#23,403 0 comments 0 reactions 1 assignee Claimed by @lystopad View on GitHub
Caplin type:feature
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Caplin decides every proposal between the local EL payload and an MEV relay bid, but that decision is only visible in logs. There is no metric, so an operator cannot chart or alert on how their proposals split between the two.

I enumerated every metric registered in the tree (443 names on `main`). Nothing counts proposals by source. The closest is `block_producer_delay{type="production"}` (`execution/metrics/block.go:36`), whose Prometheus `_count` gives local EL payload *builds* — but it increments even when the relay bid wins, so it cannot express the split.

## Why this matters for validator nodes

Choosing local execution when a relay bid was several times larger costs the operator real money, silently. #14883 is exactly that: `builderValue` was ~19x `execValue`, `boostFactor=0` made Caplin pick local anyway, and the operator only discovered it by reading logs after the block was already proposed.

Today the evidence exists only as log lines:

- `cl/beacon/handler/block_production.go:889` — `Check mev bid` with `useLocalExec`, `execValue`, `builderValue`, `boostFactor`
- `cl/beacon/handler/block_production.go:672` — `BlockProduction: Block produced` with `blinded`
- `cl/beacon/handler/block_production.go:1676` — `Successfully submitted blinded block`

Proposals are rare, so a regression here can sit unnoticed for weeks. A counter makes it a dashboard panel and an alert instead of an archaeology exercise.

## Proposed metric

One counter, two label values, incremented at the `useLocalExec` branch:

```
block_production_source{source="local"}
block_production_source{source="builder"}
```

Suggested second counter, because the two genuinely diverge: choosing the relay is not the same as the relay block landing. If `SubmitBlindedBlocks` fails at `block_production.go:1670` the proposer misses the slot entirely — the worst outcome and the one most worth alerting on.

```
block_production_blinded_publish{result="ok"|"error"}
```

Both belong on the Caplin side (`cl/monitor/metrics.go`), since the EL never learns which payload was used.

Optional, cheap while touching this code: a gauge for the last observed `execValue` / `builderValue` pair would show *how much* was left on the table, not just which side won. #14883 would have been obvious from that alone.

## Reuse beyond Caplin

Worth designing with these in mind rather than solving now:

- **External CL + mev-boost.** The gap does not apply — mev-boost and the CL export their own proposal metrics, and Erigon's EL genuinely cannot tell whether its payload was used. Nothing to add.
- **EL-side attribution (needs design).** Erigon already keeps the last built block in `LatestBlockBuiltStore` (`execution/builder/latest_block_built.go`) for `eth_getBlockByNumber("pending")`. When a block arrives whose `parentHash` and `timestamp` both match a payload we built, that is the same slot, so a differing block hash means our proposal went out through a relay. That would give a local-vs-external signal from the EL alone and work under any CL. It needs care around missed slots and reorgs before it can be trusted, so it should not block the Caplin counter above.
- **Naming.** If EL-side attribution lands later, both sources should share the `source="local"|"builder"` label vocabulary so one dashboard works either way.

## Scope

`cl/monitor/metrics.go` plus the increment sites in `cl/beacon/handler/block_production.go`. No behaviour change. Ships on `/debug/metrics/prometheus` with `--metrics`, so it needs a panel in `dashboards/erigon_custom_metrics/` to be useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.