HarperFast / HarperFast/harper

Native OpenMetrics /metrics endpoint with incremental collectors (replace polling prometheus-exporter)

Open
#1,636 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem

Harper ships **zero application metrics** today — no `harper*`/`hdb*` custom metrics exist; the only app telemetry is the log stream (333GB/day, of which the entire extracted alerting signal is two restart-detection strings). Meanwhile `hdb_analytics` already collects rich series (utilization, table/database size, RocksDB stats, per-second raw analytics), and the external `prometheus-exporter` component is a known CPU hotspot because it polls heavyweight operations (effectively `system_information`) on every scrape (prometheus-exporter#44, #45).

## Proposal

A core-native `/metrics` endpoint (operations API or built-in component) serving `application/openmetrics-text` (#376):

- **Incremental in-memory collectors** — counters/gauges updated at event time on each worker, aggregated across threads on scrape (same pattern as the status system's crossThread collector). No per-scrape system scans.
- Source existing `hdb_analytics` collectors where they're already cheap, plus the missing operational gauges:
- ops/sec by operation type; per-resource request counts (#961)
- write/read transaction queue depth (#592)
- cache hit rate / cache-hit duration (#682)
- replication per-peer: connection state, lag (applied vs head), resume-cursor position, blob in-flight (harper-pro#437 Tier 2)
- worker-thread availability ratio (#668)
- restart counter + uptime (replaces the log-string restart monitors)

## How it lands on the Grafana migration

This slots directly into the in-flight Datadog→Grafana migration ([HarperFast/observability](https://github.com/HarperFast/observability)) as one of two complementary paths:

1. **Scrape path (fleet alerting + retention):** the Grafana Alloy collectors already deployed on Fabric hosts scrape `/metrics` into Mimir — Harper app metrics land in the same stack as the existing 25 alert rules, so alert rules on replication lag or restart rate are just more Terraform in the observability repo. This path works when the instance is down/unreachable.
2. **Pull path (deep drill-down, zero export cost):** the published [grafana-datasource](https://github.com/HarperFast/grafana-datasource) plugin already queries `hdb_analytics` cluster-wide (≥5.1.13 fan-out). High-cardinality diagnostic series (per-table, per-resource, per-peer detail) stay in Harper and are queried in place. New gauges surfaced through `hdb_analytics` appear here automatically.

Rule of thumb: low-cardinality health gauges → Mimir for alerting; everything else stays queryable in place.

This supersedes the polling architecture of `prometheus-exporter` (which can become a thin shim or be deprecated for v5+).

## Related
#376, #592, #961, #682, #668, prometheus-exporter#44/#45, harper-pro#437 (Tier 2), HarperFast/observability, HarperFast/grafana-datasource.

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start by reading the status system's crossThread collector pattern and the existing hdb_analytics collectors described in the issue. Then inspect the operations API or built-in component options for serving an OpenMetrics endpoint. Done means a core-native /metrics endpoint with incremental collectors, scrape-time thread aggregation, and the listed operational metrics without per-scrape system scans.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, prometheus
Domain
api, backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.