Expose Prometheus `/metrics` endpoint from the daemon
- Dominant language
- Rust
- Stars
- 3
- Forks
- 1
- Avg merge
- 4h 50m
- Merged PRs (30d)
- 1
Description
dotagent already emits structured logs and supports OTel trace export to Honeycomb / Tempo / Jaeger / Datadog (see [`docs/guides/observability.md`](docs/guides/observability.md)). What it does not expose is a Prometheus-style metrics endpoint — the single most common observability ask for any scheduler / cron-replacement.
Today, to know how often `databricks-cost-daily` is failing this week, you have to grep the audit log. There is no scrape endpoint.
## Proposal
The daemon exposes an HTTP endpoint (off by default, opt-in via `config.toml`) that serves Prometheus text format on `GET /metrics`:
```
# HELP dotagent_agent_runs_total Total runs per (agent, schedule)
# TYPE dotagent_agent_runs_total counter
dotagent_agent_runs_total{agent="hello-fish",schedule="manual",result="success"} 12
dotagent_agent_runs_total{agent="hello-fish",schedule="manual",result="failure"} 0
# HELP dotagent_agent_duration_seconds Duration of last run
# TYPE dotagent_agent_duration_seconds gauge
dotagent_agent_duration_seconds{agent="hello-fish",schedule="manual"} 12
# HELP dotagent_agent_last_success_age_seconds Seconds since last success
# TYPE dotagent_agent_last_success_age_seconds gauge
dotagent_agent_last_success_age_seconds{agent="hello-fish",schedule="manual"} 432
# HELP dotagent_health_state Current health state (0=ok 1=degraded 2=failing 3=stale)
# TYPE dotagent_health_state gauge
dotagent_health_state{agent="hello-fish",schedule="manual"} 0
```
Config:
```toml
[metrics]
enabled = true
bind = "127.0.0.1:9091"
```
## Acceptance criteria
- [ ] `/metrics` returns 200 with valid Prometheus text format when enabled.
- [ ] Off by default (no listener, no port held).
- [ ] Counters survive daemon restart (read state from disk on boot).
- [ ] At minimum these series: runs_total (counter), duration_seconds (gauge), last_success_age_seconds (gauge), health_state (gauge), preflight_aborts_total (counter), notifications_sent_total (counter).
- [ ] Binds to localhost by default; non-localhost binds emit a warning log.
- [ ] [`docs/guides/observability.md`](docs/guides/observability.md) gains a "Prometheus metrics" section.
## Where to start
- `crates/dotagent/src/commands/daemon.rs` — daemon entry point, where the HTTP listener would spin up.
- `crates/dotagent-telemetry/` — existing telemetry crate.
- `crates/dotagent-state/src/audit.rs` — counter sources.
## Non-goals
- OpenMetrics / exemplars (text format is enough for v1).
- Pushing to remote write. Scrape only.
- Per-agent histograms (gauges + counters cover the 80% use case).
Contributor guide
Research direction
Start by reading crates/dotagent/src/commands/daemon.rs, crates/dotagent-telemetry/, and crates/dotagent-state/src/audit.rs to trace daemon startup and counter sources. Review docs/guides/observability.md for existing telemetry guidance. Done means an opt-in localhost listener serves valid /metrics output, persists counters across restart, covers the listed series, warns on non-localhost binds, and documents the feature.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, rust
- Domain
- backend, documentation, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100