avelino / avelino/dotagent

Expose Prometheus `/metrics` endpoint from the daemon

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
area: daemon area: telemetry type: feature
Dominant language
Rust
Stars
3
Forks
1
Avg merge
4h 50m
Merged PRs (30d)
1

Description

dotagent already emits structured logs and supports OTel trace export to Honeycomb / Tempo / Jaeger / Datadog (see [`docs/guides/observability.md`](docs/guides/observability.md)). What it does not expose is a Prometheus-style metrics endpoint — the single most common observability ask for any scheduler / cron-replacement.

Today, to know how often `databricks-cost-daily` is failing this week, you have to grep the audit log. There is no scrape endpoint.

## Proposal

The daemon exposes an HTTP endpoint (off by default, opt-in via `config.toml`) that serves Prometheus text format on `GET /metrics`:

```
# HELP dotagent_agent_runs_total Total runs per (agent, schedule)
# TYPE dotagent_agent_runs_total counter
dotagent_agent_runs_total{agent="hello-fish",schedule="manual",result="success"} 12
dotagent_agent_runs_total{agent="hello-fish",schedule="manual",result="failure"} 0

# HELP dotagent_agent_duration_seconds Duration of last run
# TYPE dotagent_agent_duration_seconds gauge
dotagent_agent_duration_seconds{agent="hello-fish",schedule="manual"} 12

# HELP dotagent_agent_last_success_age_seconds Seconds since last success
# TYPE dotagent_agent_last_success_age_seconds gauge
dotagent_agent_last_success_age_seconds{agent="hello-fish",schedule="manual"} 432

# HELP dotagent_health_state Current health state (0=ok 1=degraded 2=failing 3=stale)
# TYPE dotagent_health_state gauge
dotagent_health_state{agent="hello-fish",schedule="manual"} 0
```

Config:

```toml
[metrics]
enabled = true
bind = "127.0.0.1:9091"
```

## Acceptance criteria

- [ ] `/metrics` returns 200 with valid Prometheus text format when enabled.
- [ ] Off by default (no listener, no port held).
- [ ] Counters survive daemon restart (read state from disk on boot).
- [ ] At minimum these series: runs_total (counter), duration_seconds (gauge), last_success_age_seconds (gauge), health_state (gauge), preflight_aborts_total (counter), notifications_sent_total (counter).
- [ ] Binds to localhost by default; non-localhost binds emit a warning log.
- [ ] [`docs/guides/observability.md`](docs/guides/observability.md) gains a "Prometheus metrics" section.

## Where to start

- `crates/dotagent/src/commands/daemon.rs` — daemon entry point, where the HTTP listener would spin up.
- `crates/dotagent-telemetry/` — existing telemetry crate.
- `crates/dotagent-state/src/audit.rs` — counter sources.

## Non-goals

- OpenMetrics / exemplars (text format is enough for v1).
- Pushing to remote write. Scrape only.
- Per-agent histograms (gauges + counters cover the 80% use case).

Contributor guide

Open the contributing guide

Research direction

Start by reading crates/dotagent/src/commands/daemon.rs, crates/dotagent-telemetry/, and crates/dotagent-state/src/audit.rs to trace daemon startup and counter sources. Review docs/guides/observability.md for existing telemetry guidance. Done means an opt-in localhost listener serves valid /metrics output, persists counters across restart, covers the listed series, warns on non-localhost binds, and documents the feature.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, rust
Domain
backend, documentation, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.