PeerDAS Metrics Revamp: Fix phantom metrics, add missing dashboard panels, improve observability

# PeerDAS Metrics Revamp Proposal

## Executive Summary

Deep analysis of Lodestar's PeerDAS metrics reveals:
1. **Dashboard has phantom metrics** — 3 metrics referenced that don't exist
2. **Existing metrics not dashboarded** — 6+ metrics defined but not visible to operators
3. **Gaps vs other clients** — Missing KZG timing, reconstruction tracking, peer-per-subnet
4. **Operator questions unanswered** — Key health metrics missing

This proposal outlines a complete revamp to make the `lodestar_peerdas.json` dashboard sane and comprehensive.

---

## Part 1: Fix Dashboard Phantom Metrics

These metrics are referenced in `dashboards/lodestar_peerdas.json` but don't exist in code:

### 1.1 `beacon_custody_groups_backfilled`

**Problem:** Dashboard panel "Backfilled custody groups" shows nothing.

**Solution:** Add gauge to track custody backfill progress.

```typescript
// In beacon.ts dataColumn section
custodyGroupsBackfilled: register.gauge({
name: "beacon_custody_groups_backfilled",
help: "Number of custody groups fully backfilled",
}),
```

**Instrumentation:** Increment when a custody group completes backfill sync.

### 1.2 `beacon_data_availability_reconstruction_time_seconds`

**Problem:** Dashboard panel "Time taken to reconstruct columns" shows nothing.

**Solution:** This exists as `lodestar_recover_data_column_sidecar_recover_time_seconds` - dashboard should use the correct name, or we should add an alias.

**Fix:** Update dashboard to use existing metric, OR add:
```typescript
dataAvailabilityReconstructionTime: register.histogram({
name: "beacon_data_availability_reconstruction_time_seconds",
help: "Time to reconstruct data columns from available subset",
buckets: [0.1, 0.25, 0.5, 1, 2, 5, 10],
}),
```

### 1.3 `lodestar_gossip_data_column_received_to_gossip_validate_seconds`

**Problem:** Dashboard panel "Data column recv to gossip validation delay" shows nothing.

**Solution:** Add histogram tracking gossip validation latency:
```typescript
// In lodestar.ts gossip section
dataColumnReceivedToGossipValidate: register.histogram({
name: "lodestar_gossip_data_column_received_to_gossip_validate_seconds",
help: "Time from data column receipt to gossip validation completion",
buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1],
}),
```

---

## Part 2: Add Missing Metrics to Dashboard

These exist but aren't in the dashboard:

### 2.1 KZG Proofs Verification Time (CRITICAL)

**Metric:** `beacon_data_column_sidecar_kzg_proofs_verification_seconds`

**Why important:** KZG verification is the CPU bottleneck. Operators need to see this.

**Add to dashboard:** New panel in "Computation, verification, custody" section:
```
Title: "KZG proofs verification time (p99)"
Expr: histogram_quantile(0.99, rate(beacon_data_column_sidecar_kzg_proofs_verification_seconds_bucket[$rate_interval]))
```

### 2.2 Missing Custody Columns (CRITICAL)

**Metric:** `lodestar_data_columns_missing_custody_columns_count`

**Why important:** Direct indicator of data availability failure.

**Add to dashboard:** New panel in "Health" section (create if needed):
```
Title: "Missing custody columns"
Expr: lodestar_data_columns_missing_custody_columns_count
Alert: > 0 for > 1 minute
```

### 2.3 Processing Skip Count

**Metric:** `beacon_data_column_sidecar_processing_skip_total`

**Why important:** Shows how many columns are skipped (duplicates, etc.)

**Add to dashboard:** Add as third line in "Data column sidecars gossip verification":
```
Expr: rate(beacon_data_column_sidecar_processing_skip_total[$rate_interval]) * 12
Legend: "skipped"
```

### 2.4 Sent Peers Per Subnet

**Metric:** `lodestar_data_column_sent_peers_per_subnet`

**Why important:** Network contribution / good citizen metric.

**Add to dashboard:** New panel in "Network" section:
```
Title: "Peers served per subnet"
Expr: lodestar_data_column_sent_peers_per_subnet
```

### 2.5 Engine Result

**Metric:** `lodestar_data_column_engine_result_total`

**Why important:** Tracks EL interaction success/failure.

**Add to dashboard:** New panel near getBlobsV2 section.

---

## Part 3: New Metrics to Add (Gaps vs Other Clients)

Based on Lighthouse/Prysm comparison:

### 3.1 Reconstruction Failure Tracking

```typescript
reconstructionFailures: register.counter<{reason: string}>({
name: "beacon_data_availability_reconstruction_failures_total",
help: "Failed data column reconstructions",
labelNames: ["reason"], // "insufficient_columns", "computation_error"
}),

reconstructionColumnsAtTrigger: register.histogram({
name: "beacon_data_availability_reconstruction_columns_available",
help: "Number of columns available when reconstruction triggered",
buckets: [32, 48, 64, 80, 96, 112, 128],
}),
```

### 3.2 Per-Custody-Group Peer Count

```typescript
custodyGroupPeerCount: register.gauge<{custodyGroup: string}>({
name: "beacon_custody_group_peer_count",
help: "Number of peers serving each custody group",
labelNames: ["custodyGroup"],
}),
```

**Why critical:** #1 operator concern is "do I have peers for my custody groups?"

### 3.3 Column Source Breakdown

```typescript
columnsReceived: register.counter<{source: string}>({
name: "beacon_data_columns_received_total",
help: "Data columns received by source",
labelNames: ["source"], // "gossip", "req_resp", "reconstruction", "engine"
}),
```

**Why important:** Healthy = mostly gossip. High fetch = problem.

### 3.4 RPC Latency

```typescript
dataColumnsByRangeLatency: register.histogram({
name: "beacon_rpc_data_columns_by_range_response_seconds",
help: "Latency of DataColumnSidecarsByRange responses",
buckets: [0.1, 0.25, 0.5, 1, 2, 5, 10],
}),
```

### 3.5 Time to Data Availability

```typescript
timeToDataAvailability: register.histogram({
name: "beacon_time_to_data_availability_seconds",
help: "Time from block receipt to data availability confirmed",
buckets: [0.5, 1, 2, 3, 4, 5, 6, 8, 10, 12],
}),
```

**Why critical:** End-to-end health metric.

---

## Part 4: Dashboard Restructure

### Proposed Sections

1. **Health Overview** (NEW)
- Custody group count (stat)
- Custody group peer counts (bar chart)
- Missing columns alert (stat, red when > 0)
- Time to DA (gauge)

2. **Network** (existing, enhanced)
- Data columns receiving delay (existing)
- Data columns source breakdown (new)
- Peers per custody group (new)

3. **Gossip Verification** (existing)
- Processing requests/success/skip
- Gossip errors by type
- Verification latency

4. **engine_getBlobsV2** (existing)
- Requests & responses
- Duration
- Error rate

5. **KZG & Computation** (enhanced)
- Column computation time (existing)
- KZG verification time (ADD)
- Inclusion proof verification (existing)

6. **Reconstruction** (existing, enhanced)
- Reconstruction time (FIX metric name)
- Columns before reconstruction
- Reconstruction result
- Reconstruction failures by reason (ADD)

7. **Custody & Backfill** (NEW section)
- Custody groups (existing stat)
- Backfilled groups (FIX phantom)
- Backfill progress over time

---

## Part 5: Implementation Plan

### Phase 1: Fix Phantom Metrics (P0)
- [ ] Fix dashboard to use correct metric names OR add missing metrics
- [ ] Delete references to non-existent metrics
- Effort: Small (1-2 hours)

### Phase 2: Add Existing Metrics to Dashboard (P1)
- [ ] Add KZG verification time panel
- [ ] Add missing custody columns panel
- [ ] Add skip count to verification graph
- Effort: Small (1-2 hours)

### Phase 3: Add New Critical Metrics (P1)
- [ ] Add custody group peer count metric
- [ ] Add time-to-DA metric
- [ ] Add column source breakdown
- Effort: Medium (4-8 hours)

### Phase 4: Dashboard Restructure (P2)
- [ ] Add Health Overview section
- [ ] Reorganize for operator workflow
- Effort: Medium (2-4 hours)

### Phase 5: Parity with Other Clients (P3)
- [ ] Add reconstruction failure tracking
- [ ] Add RPC latency metrics
- [ ] Review against Lighthouse metrics list
- Effort: Large (1-2 days)

---

## Appendix A: Full Metric Inventory

### beacon.ts dataColumn (12 metrics)

| Metric | Used in Dashboard | Status |
|--------|-------------------|--------|
| `beacon_data_column_sidecar_processing_requests_total` | ✅ | OK |
| `beacon_data_column_sidecar_processing_skip_total` | ❌ | ADD |
| `beacon_data_column_sidecar_processing_successes_total` | ✅ | OK |
| `beacon_data_column_sidecar_gossip_verification_seconds` | ✅ | OK |
| `beacon_data_column_sidecar_computation_seconds` | ✅ | OK |
| `beacon_data_column_sidecar_inclusion_proof_verification_seconds` | ✅ | OK |
| `beacon_data_column_sidecar_kzg_proofs_verification_seconds` | ❌ | ADD |
| `beacon_engine_getBlobsV2_buffer_preallocation_duration_seconds` | ❌ | Optional |
| `beacon_engine_getBlobsV2_requests_total` | ✅ | OK |
| `beacon_engine_getBlobsV2_responses_total` | ✅ | OK |
| `beacon_engine_getBlobsV2_request_duration_seconds` | ✅ | OK |
| `beacon_target_custody_group_count` | ✅ | OK |

### lodestar.ts PeerDAS (9 metrics)

| Metric | Used in Dashboard | Status |
|--------|-------------------|--------|
| `lodestar_recover_data_column_sidecar_recover_time_seconds` | ❌ | ADD (or fix phantom) |
| `lodestar_data_columns_in_custody_before_reconstruction` | ✅ | OK |
| `lodestar_recover_data_column_sidecar_recovered_columns_total` | ❌ | Consider |
| `lodestar_data_column_sidecars_reconstruction_result` | ✅ | OK |
| `lodestar_data_columns_by_source` | ✅ | OK |
| `lodestar_data_column_elapsed_time_till_received_seconds` | ✅ | OK |
| `lodestar_data_column_sent_peers_per_subnet` | ❌ | ADD |
| `lodestar_data_columns_missing_custody_columns_count` | ❌ | ADD (critical!) |
| `lodestar_data_column_engine_result_total` | ❌ | Consider |

### Dashboard Phantom Metrics (3)

| Metric | Fix |
|--------|-----|
| `beacon_custody_groups_backfilled` | CREATE |
| `beacon_data_availability_reconstruction_time_seconds` | Use existing or CREATE |
| `lodestar_gossip_data_column_received_to_gossip_validate_seconds` | CREATE |

---

## Appendix B: Operator Priority Questions

Ranked by criticality:

1. **Do I have enough peers for my custody groups?** — Need per-group peer metric
2. **Am I missing data columns I should have?** — Have metric, not dashboarded
3. **Are data columns failing verification?** — Partial (need error reasons)
4. **Is my node participating in PeerDAS correctly?** — Need health indicator
5. **Gossip vs fetch ratio** — Have, could improve labels
6. **getBlobsV2 error rate** — Can derive from existing
7. **Column verification latency (p99)** — Have gossip, need KZG
8. **Bandwidth usage** — Not currently tracked
9. **Reconstruction rate** — Have
10. **Columns served to others** — Have, not dashboarded

---

## Appendix C: Code Trace (Full Inventory)

Base URL: https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src

### beacon.ts dataColumn section (12 metrics)

| # | Metric | Status | Location | Trigger |
|---|--------|--------|----------|---------|
| 1 | `beacon_data_column_sidecar_processing_requests_total` | ✅ USED | [network/processor/gossipHandlers.ts#L294](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/processor/gossipHandlers.ts#L294) | Data column submitted for gossip processing |
| 2 | `beacon_data_column_sidecar_processing_skip_total` | ✅ USED | [network/processor/gossipHandlers.ts#L301](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/processor/gossipHandlers.ts#L301) | Processing skipped (already known/validated) |
| 3 | `beacon_data_column_sidecar_processing_successes_total` | ✅ USED | [network/processor/gossipHandlers.ts#L349](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/processor/gossipHandlers.ts#L349) | Passes gossip verification |
| 4 | `beacon_data_column_sidecar_gossip_verification_seconds` | ✅ USED | [network/processor/gossipHandlers.ts#L331](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/processor/gossipHandlers.ts#L331) | Full gossip verification time |
| 5 | `beacon_data_column_sidecar_computation_seconds` | ✅ USED | [api/impl/beacon/blocks/index.ts#L106](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/api/impl/beacon/blocks/index.ts#L106) | Computing sidecars from block/blobs |
| 6 | `beacon_data_column_sidecar_inclusion_proof_verification_seconds` | ✅ USED | [chain/validation/dataColumnSidecar.ts#L169](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/validation/dataColumnSidecar.ts#L169) | Inclusion proof verification |
| 7 | `beacon_data_column_sidecar_kzg_proofs_verification_seconds` | ✅ USED | [chain/validation/dataColumnSidecar.ts#L181](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/validation/dataColumnSidecar.ts#L181) | KZG proof verification |
| 8 | `beacon_engine_getBlobsV2_buffer_preallocation_duration_seconds` | ✅ USED | [chain/GetBlobsTracker.ts#L77](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/GetBlobsTracker.ts#L77) | Buffer pre-allocation |
| 9 | `beacon_engine_getBlobsV2_requests_total` | ✅ USED | [util/execution.ts#L155](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/util/execution.ts#L155) | getBlobsV2 request sent |
| 10 | `beacon_engine_getBlobsV2_responses_total` | ✅ USED | [util/execution.ts#L168](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/util/execution.ts#L168) | Successful response received |
| 11 | `beacon_engine_getBlobsV2_request_duration_seconds` | ✅ USED | [util/execution.ts#L156](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/util/execution.ts#L156) | Request/response roundtrip |
| 12 | `beacon_target_custody_group_count` | ✅ USED | [chain/chain.ts#L275](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/chain.ts#L275) | Chain init / custody update |

### lodestar.ts PeerDAS metrics (9 metrics)

| # | Metric | Status | Location | Trigger |
|---|--------|--------|----------|---------|
| 13 | `lodestar_recover_data_column_sidecar_recover_time_seconds` | ✅ USED | [util/dataColumns.ts#L379](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/util/dataColumns.ts#L379) | During reconstruction |
| 14 | `lodestar_data_columns_in_custody_before_reconstruction` | ✅ USED | [util/dataColumns.ts#L369](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/util/dataColumns.ts#L369) | Before reconstruction attempt |
| 15 | `lodestar_recover_data_column_sidecar_recovered_columns_total` | ⚠️ **UNUSED** | [metrics/metrics/lodestar.ts#L815](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/metrics/metrics/lodestar.ts#L815) | **Never incremented** |
| 16 | `lodestar_data_column_sidecars_reconstruction_result` | ✅ USED | [chain/ColumnReconstructionTracker.ts#L72](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/ColumnReconstructionTracker.ts#L72) | After reconstruction (success/failure) |
| 17 | `lodestar_data_columns_by_source` | ✅ USED | [api/impl/beacon/blocks/index.ts#L329](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/api/impl/beacon/blocks/index.ts#L329) | Columns received/created |
| 18 | `lodestar_data_column_elapsed_time_till_received_seconds` | ✅ USED | [network/processor/gossipHandlers.ts#L578](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/processor/gossipHandlers.ts#L578) | Delay from slot start to receipt |
| 19 | `lodestar_data_column_sent_peers_per_subnet` | ✅ USED | [api/impl/beacon/blocks/index.ts#L311](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/api/impl/beacon/blocks/index.ts#L311) | Publishing sidecars |
| 20 | `lodestar_data_columns_missing_custody_columns_count` | ✅ USED | [network/reqresp/utils/dataColumnResponseValidation.ts#L62](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/reqresp/utils/dataColumnResponseValidation.ts#L62) | Custody columns missing from DB |
| 21 | `lodestar_data_column_engine_result_total` | ✅ USED | [chain/GetBlobsTracker.ts#L102](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/GetBlobsTracker.ts#L102) | After sending to EL |

### Additional lodestar.ts metrics found

| # | Metric | Status | Location | Trigger |
|---|--------|--------|----------|---------|
| 22 | `lodestar_import_columns_by_source_total` | ✅ USED | [chain/blocks/importBlock.ts#L525](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/blocks/importBlock.ts#L525) | Columns imported during block import |
| 23 | `lodestar_seen_block_input_cache_duplicate_column_count` | ✅ USED | [chain/seenCache/seenGossipBlockInput.ts#L325](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/seenCache/seenGossipBlockInput.ts#L325) | Duplicate column seen |

### network/core/metrics.ts PeerDAS metrics

| # | Metric | Status | Location | Trigger |
|---|--------|--------|----------|---------|
| 24 | `lodestar_peer_column_group_count` | ✅ USED | [network/peers/peerManager.ts#L874](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/peers/peerManager.ts#L874) | Peer metrics scrape |
| 25 | `lodestar_peer_count_per_sampling_group` | ✅ USED | [network/peers/utils/prioritizePeers.ts#L311](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/peers/utils/prioritizePeers.ts#L311) | Peer prioritization |
| 26 | `lodestar_discovery_custody_group_peers_to_connect` | ✅ USED | [network/peers/discover.ts#L170](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/peers/discover.ts#L170) | Discovery |
| 27 | `lodestar_discovery_custody_groups_to_connect` | ✅ USED | [network/peers/discover.ts#L171](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/peers/discover.ts#L171) | Discovery |
| 28 | `lodestar_sync_head_sync_peers_count` | ✅ USED | [sync/range/chain.ts#L168](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/sync/range/chain.ts#L168) | Head sync peer count |
| 29 | `lodestar_sync_finalized_sync_peers_count` | ✅ USED | [sync/range/chain.ts#L665](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/sync/range/chain.ts#L665) | Finalized sync peer count |

### network/gossip/metrics.ts PeerDAS metrics

| # | Metric | Status | Location | Trigger |
|---|--------|--------|----------|---------|
| 30 | `lodestar_gossip_mesh_peers_by_data_column_subnet_count` | ✅ USED | [network/gossip/gossipsub.ts#L256](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/gossip/gossipsub.ts#L256) | Gossip metrics scrape |
| 31 | `lodestar_gossip_topic_peers_by_data_column_subnet_count` | ✅ USED | [network/gossip/gossipsub.ts#L256](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/network/gossip/gossipsub.ts#L256) | Gossip metrics scrape |

### Summary

| Category | Count |
|----------|-------|
| **Total PeerDAS metrics** | 31 |
| **Used** | 30 |
| **Unused (dead code)** | 1 |

**Dead code:** `lodestar_recover_data_column_sidecar_recovered_columns_total` is defined but never incremented. Should track columns recovered during reconstruction.

---

## References

- PeerDAS spec analysis (das-core.md, p2p-interface.md, validator.md, fork-choice.md)
- Lighthouse/Prysm/Nimbus/Teku metrics comparison
- Operator workflow analysis

Start with dashboards/lodestar_peerdas.json and compare its metric references with the inventories in beacon.ts and lodestar.ts. Trace existing instrumentation through network/processor/gossipHandlers.ts, then agree on a phase and scope before changing anything. Done means the selected metrics and dashboard panels use valid names and cover the agreed operator views.

Written by the indexing model from the issue text.