evstack / evstack/apex

Observability: metrics, logging, and tracing

Open
#7 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

Implement observability infrastructure for apex: Prometheus metrics, structured logging, and OpenTelemetry tracing.

## Metrics

Instrument all subsystems with OTel metrics exposed via Prometheus endpoint.

### Sync engine
- `apex_sync_head` (gauge) — last synced height
- `apex_sync_network_head` (gauge) — upstream network head
- `apex_sync_lag_seconds` (gauge) — time behind network head
- `apex_sync_backfill_duration` (histogram) — per-batch backfill latency
- `apex_sync_errors_total` (counter) — sync errors by type
- `apex_sync_backfill_progress_pct` (gauge) — backfill completion percentage

### API
- `apex_rpc_request_duration` (histogram) — per-method latency
- `apex_rpc_request_total` (counter) — per-method call count
- `apex_rpc_errors_total` (counter) — per-method errors

### Store
- `apex_store_query_duration` (histogram) — SQLite query latency
- `apex_store_insert_duration` (histogram) — insert latency
- `apex_store_size_bytes` (gauge) — DB file size

### Subscriptions
- `apex_subscriptions_active` (gauge) — active subscription count
- `apex_subscription_deliveries` (counter) — messages delivered
- `apex_subscription_drops` (counter) — messages dropped (slow reader)

### Node
- `apex_build_info` (gauge) — with version labels
- `apex_uptime_seconds` (counter)

## Logging

- Use `slog` (Go stdlib) — no heavy dependencies like `ipfs/go-log`
- Structured key-value pairs: height, namespace, method, duration, error
- Configurable log level at startup (and ideally at runtime via admin endpoint)

## Tracing

- OpenTelemetry spans for sync fetch, store operations, and RPC handlers
- Configurable exporter (stdout for dev, OTLP for production)

## API endpoint

Expose a unified `SyncStatus()` endpoint returning current synced height, network head, sync state (backfilling/streaming), lag, and active subscriptions in one call. celestia-node spreads this across 4 separate modules.

## Configuration

```toml
[observability]
metrics_address = "0.0.0.0:9090" # Prometheus endpoint
log_level = "info" # debug, info, warn, error
tracing_enabled = false
tracing_endpoint = "" # OTLP endpoint
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.