Metrics v2: typed, dimensioned metrics with packed time-series storage

# Metrics v2: typed, dimensioned metrics with packed time-series storage

**TL;DR** — Today's analytics pipeline writes, re-reads, and main-thread-decodes every
thread's metric report every second (measured 64 KB per thread-second at 200 active
series — that extrapolates to tens of GB/day of disk churn on a busy multi-threaded
node) to produce percentiles that can be **88% wrong** the moment a dashboard spans
more than one aggregate window — and its API can't express a gauge, a distinct count,
or a fourth dimension. Proposal: **metrics v2**, a standalone typed engine (counters,
gauges, DDSketch histograms, HyperLogLog distinct counts, observables — all with
arbitrary named dimensions) over packed time-series storage. Measured against today:
**10× cheaper hot path, 12× less event-loop blocking, zero raw-table churn, ~100× less
storage for a year of retention, and cross-window percentiles become exact.** A
working, benchmarked implementation already exists; it lands as small inert PRs behind
a config kill switch, nothing breaks during migration, and the old system is deleted
only when the last metric has moved. Numbers, tradeoffs, and the rollout plan below.

## Why the current analytics pipeline is a problem

Every Harper node runs this pipeline today (`resources/analytics/`):

```
recordAction(value, metric, path?, method?, type?)
└─ per-thread Map; numeric metrics append EVERY raw value to a growing Float32Array
every 1s: sort every array, build percentile distributions,
persist the whole report to hdb_raw_analytics (disk)
every 60s (main thread): re-read + msgpack-decode all raw reports,
re-aggregate, write per-series rows to
hdb_analytics (audit: true + 3 secondary indexes)
```

Each stage has costs we have already paid for in production:

**1. The raw-table round trip stalls the main thread.** Every thread's report is
persisted to `hdb_raw_analytics` once per second, then re-read and msgpack-decoded on
the **main thread of the same process** during aggregation — the exact cost flagged in
#1538 as starving TLS handshakes. Report size scales with active series: measured
~64 KB per thread-second at 200 active numeric series, which **extrapolates to
~90 GB/day of write + decode traffic** on a 16-thread node under that load (a lighter
node with 20 active series still churns ~9 GB/day) — all through a table whose only
reader is the aggregator one thread over.

**2. Every worker sorts every second.** Numeric metrics buffer every raw observation and
sort per tick: measured **5.6 ms of event-loop blocking per second per worker** at
200 series × 1k obs/s (0.6% of every worker, and the sort is O(n log n) in observation
rate). Per-series memory scales linearly with rate — a series at 50k obs/s holds
~200–400 KB of raw floats at the end of every one-second window, where a DDSketch holds
**~5 KB at any rate**, bounded by construction.

**3. There is no valid way to combine percentiles across windows.** Stored rows carry
only point-in-time percentile fields (`p1…p999`). Any consumer that wants a percentile
over a longer range — a one-hour dashboard panel built from 60 one-minute rows, a
cluster view merging rows from the `replicated` fan-out — can only average them
(count-weighted at best), and averaging percentiles is statistically invalid. The error
is not small. Combine one calm minute (9k requests, 1–10 ms) with one slow burst
(1k requests, 200–1000 ms):

| | p99 over both windows |
|---|---|
| true | **925 ms** |
| count-weighted average of the two rows' p99s (the best stored rows allow) | **108 ms — 88% low** |
| sketch merge (proposed) | **925 ms — 0.0% off** |

An operator reading a range wider than one aggregate period would conclude the burst
never happened. The read path has no answer: `coalesce_time` relabels rows to a shared
timestamp without merging their values. In v2, every stored histogram block carries its
sketch, and sketch merges are exact — any window, any step, any set of nodes.

**4. Storage pays for schema on every row, plus audit and indexes nobody uses.** Every
aggregate row repeats attribute names, the metric name, and dimension strings — the
string `"transaction-commit-time"` is stored ~525,000 times per node per year. Every row
also writes an audit entry (hardcoded `audit: true`, even though `analytics.replicate`
defaults to false) and maintains `path`/`method`/`type` secondary indexes that reads
don't need (dimension conditions are dynamic-attribute post-filters). Reads got their own
incident: the planner's flat range estimate on `hdb_analytics` decoded a metric's entire
history instead of the requested window (#1796) and needed a forced-execution-order
workaround.

**5. Cardinality is unbounded.** Any dimension with unbounded values (a REST path with
IDs in it) creates unbounded series — unbounded memory, rows, and index entries. There
is no cap and no signal that it is happening.

**6. The API can't express what we measure.** One positional signature,
`recordAction(value, metric, path?, method?, type?)`, where the *type of the value*
picks the semantics (number → distribution, boolean → counter, function → callback).
There are no gauges (min/max/last are lost) and no distinct counts.

The three dimension slots aren't just fixed in number — they are **HTTP vocabulary
baked into a system-wide API**, and every non-HTTP subsystem puns its own concepts into
them. Today, `path` variously means a URL route (`http.ts`), a **table name**
(`db-read`/`db-write`/`db-message` in `Table.ts`/`RecordEncoder.ts`), an **MQTT topic
prefix** (`mqtt.ts`), a **model backend name** (`Models.ts`), and a numeric **listener
port** (`tls-handshake`); `method` variously means an HTTP verb, a packed MQTT verb
string like `publish,qos=1`, or `connect`/`disconnect`; `type` is a grab-bag
(`cache-hit`, `'mqtt'`, `'ws'`, `'operation'`). The names survive all the way into
stored rows and `get_analytics` output, so a consumer filtering the `db-write` metric
has to know to query `path` for a table name. Anything that genuinely needs a fourth
dimension can't have one: MQTT/WS connection gauges are faked by mutating the listener
callback's array, and `Models.ts` narrows the emitter type to hide the slots it can't
use. In v2, each metric declares its own dimension names — `{ table }`, `{ topic,
verb, qos }`, `{ backend }`, `{ port }` — and stored data says what it means.

## Proposal

A standalone, fully-optimized **metrics** engine in `resources/metrics/` — built for how
metrics systems actually work (Prometheus/OpenTelemetry semantics), not shaped by the
old storage. The old system is untouched; call sites migrate metric-by-metric; the old
pipeline is deleted when empty. The full design doc (`docs/metrics-v2.md`) lands with
the first implementation PR.

The name changes deliberately: "analytics" suggests analyzing user data; this subsystem
is operational telemetry, and *metrics* is the industry term for it (Prometheus,
OpenTelemetry, StatsD all agree). It also keeps the two systems lexically disjoint —
`metrics.*` config, `hdb_metrics*` tables, `get_metrics` — so migration state stays
greppable and nothing legacy is ambiguous.

Nothing breaks while this lands: `get_analytics`, `list_metrics`, `describe_metric`,
`server.recordAnalytics`, `onAnalyticsAggregate`, and the `analytics:` config section
keep working untouched until the final retirement step. Deployments using
`analytics.replicate` map onto v2's model as follows: metrics tables never replicate;
cluster-wide reads use the same peer fan-out `get_analytics` already uses for
non-replicated tables, and fleet-level centralization is what the exporters are for.

**DX** — declare once, record with types; dimensions are named and arbitrary:

```ts
// before
recordAction(elapsed, 'duration', handlerPath, method, 'cache-hit');
recordAction(true, 'success', handlerPath, method);

// after
const duration = metrics.histogram('http-request-duration', {
unit: 'ms',
dimensions: ['path', 'method', 'cache'],
});
duration.record(elapsed, { path, method, cache: 'hit' });

// hot paths bind the series once — recording is O(1), zero allocation
const series = duration.with({ path: '/api/dogs', method: 'GET', cache: 'hit' });
series.record(elapsed);
```

Five kinds with explicit semantics — three of which are **capabilities Harper doesn't
have today**, not just better spellings of existing ones:

```ts
// gauge — NEW: point-in-time values with real last/min/max/mean per window.
// Today a queue depth recorded as a number becomes a "distribution" and the
// spike you cared about dissolves into percentiles of samples.
const depth = metrics.gauge('write-queue-depth', { dimensions: ['database'] });
depth.set(queue.length, { database });

// cardinality — NEW: distinct counts (unique users, IPs, sessions) via HyperLogLog,
// mergeable across threads/windows/nodes. No way to express this today at all.
const users = metrics.cardinality('active-users', { dimensions: ['realm'] });
users.add(userId, { realm });

// observable — NEW as a first-class kind: sampled once per window from a callback,
// with declared cross-thread reduction (sum per-thread values vs. take one).
// Replaces today's two escape hatches (the function-valued recordAction and the
// mutate-the-listener-array trick for mqtt/ws connection counts).
metrics.observable('memory', () => process.memoryUsage(), { perThread: 'sum' });

// counter and histogram round out the set (today's booleans and numbers, typed)
const bytesOut = metrics.counter('bytes-sent', { unit: 'bytes', dimensions: ['protocol'] });
```

`histogram` is DDSketch-backed (~1% relative error, exactly mergeable); `gauge` keeps
last/min/max/sum/count per window. Redefining a metric under a different kind throws;
dimension keys not declared for the metric are ignored rather than silently minting new
series. Components get the same API via `server.metrics`.

**Built-in metrics stop being take-it-or-leave-it.** Because core metrics (the HTTP
flow, DB reads/writes, MQTT/WS) are registry-declared with named dimensions instead of
a sealed positional signature, customizing them becomes a data question rather than an
API redesign: deployments can tune which declared dimensions a built-in records (add
`status` where it matters, drop `path` where routes are unbounded), and built-in series
can gain deployment-defined context — `tenant`, `app`, `shard` — so multi-tenant
operators slice core metrics by their own vocabulary, bounded by the same series cap.
Not in the initial stages below; noted because the registry model is what makes it
possible at all.

**Engine** — per-thread `Float64Array` slab accumulators (a bound counter increment is
two array writes); one-second snapshots whose steady state transfers **only numbers**
(series definitions cross the thread boundary once, ever); main-thread merge by integer
series id; no raw table, no per-second sorts, no main-thread decode.

**Storage** — one packed block per metric per period (`hdb_metrics`), a series
dictionary that stores each dimension string once per series lifetime
(`hdb_metrics_meta`), no secondary indexes, no audit. Hour/day rollups merge sketches
exactly (with a persisted catch-up marker across restarts) and give per-resolution
retention. Reads are a single contiguous prefix scan; dimension filters resolve against
the in-memory dictionary before touching a block.

**Reads** — a `get_metrics` operation (the MCP schema already advertises this name,
wired to nothing): epoch or ISO times, step bucketing with exact sketch merges,
dimension filters, per-series or totaled output, and a fast path that serves quantiles
from precomputed fields without decoding a sketch. Cluster-wide reads keep the
`replicated` fan-out model. The flush hook (`onMetricsFlush`) is the seam for
Prometheus scrape / OTLP push exporters (DDSketch maps 1:1 onto OTLP
`ExponentialHistogram` and Prometheus native histograms) — export mode lets a fleet
ship metrics to an external store and keep local storage minimal.

## Measured (implementation exists and is benchmarked)

Apple M-series, Node 24, 2M-op runs; the benchmark script
(`unitTests/resources/metrics/engines.bench.mjs`) ships with the implementation so the
numbers are reproducible. v1 numbers run the same accumulate/sort logic as
`analytics/write.ts` verbatim.

**Hot path (one observation into an existing series):**

| operation | ns/op |
|---|---|
| v1 `recordAction(number)` (key build + map + append) | 73.2 |
| v1 `recordAction(boolean)` (key build + map + count) | 6.7 |
| **v2 bound `histogram.record()`** | **7.4** |
| **v2 bound `counter.add()`** | **1.8** |
| v2 general form `counter.add(n, {dims})` | 102.8 |

A bound histogram observation is **10× cheaper** than today's numeric `recordAction`
and costs what a boolean counter costs today; counters get ~4× cheaper. One honest
asymmetry: the general dimensions-object form costs more than today's positional call
(103 vs 73 ns) — it exists for convenience and low-rate call sites. Migration guidance
for hot v1 call sites is therefore always the bound form, which is what the migration
PRs will use.

**Per-second, per-worker snapshot (200 series × 1k obs):** 5.61 ms (sort + distribution
build) → **0.47 ms** (serialize sketches) — 12× less event-loop blocking, and the
payload becomes transferable buffers instead of object graphs.

**Raw-table churn:** ~64 KB per thread-second measured at 200 series (→ ~90 GB/day
extrapolated for 16 loaded threads) → **zero** (in-memory merge).

**Storage (200-series metric, per 60s period, msgpack + gzip as a proxy for engine
block compression):**

| | raw | compressed |
|---|---|---|
| v1 counter rows | 23.7 KB | — (5.9× larger raw than v2) |
| v2 counter block | 4.0 KB | — |
| v1 histogram rows | 45.9 KB | 19.7 KB (+ audit row per row today, + 3 index entries) |
| v2 histogram block | 115 KB | 33.7 KB (includes full sketches → exact rollups) |

Histogram base blocks are ~1.7× larger compressed than today's rows — that is the price
of carrying real sketches, which is what makes every number above them honest (exact
quantiles over any window, exact rollups). The totals still collapse because of
retention shape: today keeps 1-minute rows for a year; v2 keeps 3d of base + 30d of
hourly + 2y of daily rollups (so ranges older than 3 days render at hourly resolution —
which is finer than any dashboard draws them anyway). For that one 200-series histogram
metric over a year: **~10.4 GB (≈20 GB with audit) → ~195 MB steady-state, and
long-range dashboard queries scan 60–1440× fewer entries.**

**Cardinality:** per-metric series cap (default 1000) with a `dropped-series` counter —
a runaway dimension becomes an observable signal instead of unbounded memory.

## Maintainability

- Clean module boundaries, each independently unit-tested (77 tests): `registry`
(accumulation), `merge` (cross-thread), `store` (blocks/rollups/retention), `read`
(queries), `pipeline` (scheduling/wiring), plus pure `histogram`/`hll` (property
tests: merge associativity, quantile error bounds, serialization round-trips).
- No new dependencies. TypeStrip-compliant. The storage layer is injectable (unit
tests run against an in-memory store).
- The old system is not modified at all — no risk to existing dashboards during
migration, and every migration PR is a small, independently reviewable call-site swap.
- Explicit retirement checklist before the old pipeline is deleted (including
relocating `http.ts`'s `next-request-id` shared buffer off `hdb_analytics`, and
confirming license/pro consumers are off `onAnalyticsAggregate`/`get_analytics`).

## Alternatives considered

- **Fix the old pipeline in place under wire compatibility** (prototyped in July on
`perf/analytics-histogram-pipeline`): removes the raw table and sorts, but locks in
the row-per-series storage, the positional API, and the averaged-percentile output
shape it must stay byte-compatible with. The compatibility tax buys nothing once we
accept a versioned migration; that branch's proven parts (DDSketch, HLL, snapshot
pipeline) are reused here.
- **Adopt an off-the-shelf client (OpenTelemetry SDK / prom-client):** adds a
dependency on the hottest path in the server (against our dependency policy), has no
answer for Harper-local storage/queries (Studio dashboards need them), and per-event
costs are an order of magnitude above the slab design. Instead, v2 keeps the *model*
compatible (labels, exponential-bucket histograms) so OTLP/Prometheus export is a
serialization step, not an architecture.
- **Ship metrics to an external TSDB only:** unacceptable for the embedded/standalone
deployments that have no external store — but supported as v2's export mode for
fleets that want it.

## One-time cutover vs. incremental migration

Both are viable — the implementation exists either way — and the tradeoffs deserve to be
explicit rather than assumed:

| | one-time change | incremental |
|---|---|---|
| wins (raw table, sorts, storage) | immediate, all at once | arrive per migrated metric; the raw table only dies when the **last** legacy call site moves |
| review | one ~4k-line review | ~1k-line PRs, each reviewable in isolation |
| blast radius of a defect | every metric on every node, at once | contained to one stage / one metric family |
| revert | all-or-nothing | per PR, plus a permanent config kill switch |
| dual-system period | none | real: two pipelines to operate and reason about until retirement |
| total effort | lower (no coexistence overhead, no per-PR ceremony) | higher, and carries the classic strangler-fig risk of stalling half-migrated if priorities shift |

The proposal below takes the incremental shape for the parts where review quality and
blast radius matter most — the engine itself — but treats **migration pace as a dial,
not a structure**: once the engine has soaked, the call-site migration can be compressed
into a single release (the swaps are mechanical), which recovers most of the one-time
path's "wins arrive at once" property and caps the dual-system window. What we
explicitly do not want is the slow route's failure mode: an open-ended coexistence with
no owner — hence the stages below have an end state (retirement) with a checklist, not
an indefinite tail.

## Rollout: small PRs, each inert until the next, with a kill switch throughout

The engine lands as a stack of independently reviewable PRs, each under ~1k lines with
its own tests, and nothing observable changes until the wiring PR — which ships behind
config. A complete reference implementation already exists on a branch, so reviewers of
each small step can see the destination working (tests, benchmarks) before approving
the pieces.

**Stage A — inert foundations** (pure code, imported by nothing, zero runtime effect):

1. `histogram.ts` + `hll.ts` with their property-test suites (merge associativity,
quantile error bounds, serialization round-trips). Pure data structures.
2. `registry.ts` + `merge.ts` + tests — the typed API and cross-thread merge, unwired.
3. `store.ts` + `read.ts` + tests — blocks, rollups, retention, query engine. The
storage layer is injectable, so these are fully tested with no database.

**Stage B — wire it, opt-in first:**

4. `pipeline.ts` + `metrics:` config + the `get_metrics` operation. Gated by
`metrics.enabled` — proposed to default **off** for one release so we soak it on our
own fabric/dev nodes and anyone curious, then flip the default. The kill switch
remains permanently (recording becomes a no-op boolean check).
5. System metrics move to observables — no external callers, exercises every metric
kind end-to-end on every node, and gives the pipeline its first real production
proof.

**Stage C — migrate by value, one metric family per PR:**

6. HTTP request metrics first (highest volume → biggest measurable win; each migration
PR states the before/after cost for its metrics), then MQTT/WS, Table/RecordEncoder,
Models (coordinated with license enforcement). Every PR is a mechanical call-site
swap to a bound handle — small, independently revertable, and the old system keeps
serving every metric that hasn't moved.
7. Exporters (Prometheus scrape, OTLP push) behind config — any time after stage B.

**Stage D — consumers, then retirement:**

8. Dashboards/MCP move `get_analytics` → `get_metrics`; license/pro moves
`onAnalyticsAggregate` → `onMetricsFlush`.
9. Retirement checklist (relocate the `next-request-id` buffer, confirm external
consumers are off the legacy seams), then delete `resources/analytics/`.

At every point in this sequence the system is shippable, the old pipeline is intact for
anything not yet migrated, and a problem at any stage is contained to that stage's PR
or the `metrics.enabled` switch.

Tradeoffs accepted:

- Two pipelines coexist during migration (the old one's costs shrink per migrated
metric); a migrated metric's history stays in `hdb_analytics` until its retention
lapses rather than being converted.
- **Crash window widens.** v1 persists every thread's report each second, so a crash
loses ≤1 s of metrics; v2 buffers in memory and loses up to one flush period (≤60 s).
That per-second durability is exactly where the ~90 GB/day churn comes from, and for
observability data the trade is deliberate.
- **The series cap has no idle eviction yet.** A dimension that rotates values (e.g. a
token in a topic name) can exhaust a metric's 1000-series cap for the life of the
process; the `dropped-series` counter makes it visible, and idle-series eviction at
flush boundaries is planned as a fast follow.

## Status

A complete working implementation of stages A–B exists as a reference branch (engine,
store, rollups, `get_metrics`, config, 77 unit tests, benchmark suite, cross-model
reviewed) — every number above is measured from it. If this direction is agreed, it
gets split into the stage A/B PR stack rather than landed whole.

---

_Drafted by Claude (agent) with Joe; benchmark numbers measured from the reference implementation._