apache / apache/hugegraph

[Feature] Store exposes no shard recovery metrics (sync progress, raft lag) during disaster recovery

Open
#3,136 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
636
Avg merge
3d 11h
Merged PRs (30d)
14

Description

## Feature Description (功能描述)

### Problem

When a Store node is rebuilt after data loss and rejoins the cluster, its shards recover through raft snapshot install plus log catch-up. On a graph of any real size this takes minutes. Under the current implementation, that progress is visible only in log lines: no exported metric reports how many partitions are still syncing, how far along a snapshot install is, or how far the local raft log trails the leader. This makes it hard for an operator to distinguish a healthy long recovery from a stuck one, or to estimate remaining time.

Operational observation from my own fault testing (not a code claim): after deleting a Store pod and its volume in Kubernetes and letting the StatefulSet rebuild it, the only available signals were pod readiness and log lines such as `Raft {} begin loadSnapshot` / `Raft {} end loadSnapshot`. Nothing on the Prometheus endpoint moved in a way that reflected recovery progress.

### What the Store exports today (verified on master)

The Store node already runs a Spring actuator Prometheus endpoint. `hg-store-dist/src/assembly/static/conf/application.yml` (lines 22-30) enables the Prometheus meter registry and exposes all web endpoints, so `/actuator/prometheus` is served on the REST port. Meter families are registered in `hg-store-node/src/main/java/org/apache/hugegraph/store/node/metrics/MetricsConfig.java` (lines 32-46), which adds the common tag `hg=store` and initializes five sources:

- `StoreMetrics` (`store/node/metrics/StoreMetrics.java`, lines 51-79): `hg.up`, `hg.graphs`, and `hg.partitions{graph=...}` which is only a count of partitions per graph.
- `JRaftMetrics` (`store/node/metrics/JRaftMetrics.java`, lines 73-126): `jraft.groups` plus a generic re-export of whatever sofa-jraft `NodeMetrics` registers per raft group (latency timers, replicator counters, tagged `group=`). These are low level jraft internals; nothing in them is partition aware, and nothing expresses "this group is installing a snapshot" or "this group is N entries behind".
- `RocksDBMetrics` (`rocks.stats.*` prefix, `RocksDBMetricsConst.java` line 32), `ProcfsMetrics` (`process_memory.*`), `GRpcExMetrics` (`grpc.*`).

I read every file in the `store/node/metrics` package on master: there is no metric family for partition work state, shard sync state, snapshot install progress, raft log lag, leader count per store, or a recovering-partition count.

### Where recovery progress already lives in memory

The information needed for such metrics already exists in process; it is just never handed to the meter registry:

- Snapshot install: `hg-store-core/src/main/java/org/apache/hugegraph/store/raft/HgStoreStateMachine.java`, `onSnapshotLoad` (lines 216-244) knows the group id and the snapshot's last included index, and `hg-store-core/src/main/java/org/apache/hugegraph/store/snapshot/HgSnapshotHandler.java`, `onSnapshotLoad` (lines 165-203) walks the snapshot data directory while loading it into RocksDB. Start, end, and the file set (with sizes on disk) are all known there; today the only output is log lines (lines 177-180).
- Log catch-up: `HgStoreStateMachine.onApply` advances `committedIndex` on every applied entry (lines 105, 119-121), surfaced per group by `PartitionEngine.getCommittedIndex` (`hg-store-core/src/main/java/org/apache/hugegraph/store/PartitionEngine.java`, lines 718-720). The same value is already packaged per partition into `Metapb.RaftStats` for the PD heartbeat by `hg-store-core/src/main/java/org/apache/hugegraph/store/metric/HgMetricService.java`, `getRaftMetrics` (lines 114-124), so lag against the leader is computable from data the system already collects.
- Partition and shard state: `HgMetricService.getStoreMetrics` (lines 126-157) attaches each partition's `workState` and local shard role to the heartbeat, and `HeartbeatService.partitionHeartbeat` (`hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java`, lines 290-352) reports per-shard `SState_Normal` / `SState_Offline`. The proto already defines the exact states a recovery dashboard needs, including `SState_Snapshot` ("Install snapshots") in `hg-pd-grpc/src/main/proto/metapb.proto` (lines 251-259) and `PartitionState` (lines 111-121).
- Leader count: `HgStoreEngine.getLeaderPartition` (`hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java`, lines 536-544) already computes the set of groups this node leads. Raft metrics are enabled on every group (`PartitionEngine.java`, lines 199 and 244).
- There is a JSON view of some of this (`/v1/partitions` in `hg-store-node/src/main/java/org/apache/hugegraph/store/node/controller/PartitionAPI.java`, lines 66-113, with leader, term, committed index, work state), but it is an ad hoc REST endpoint, not something Prometheus scrapes or alerts on.

On the PD side, `StoreNodeService.heartBeat` (`hg-pd-core/src/main/java/org/apache/hugegraph/pd/StoreNodeService.java`, lines 722-784) persists all of this per store, and `TaskScheduleService.patrolStores` (`hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java`, lines 121-129 and 202-229) patrols store liveness every 60 seconds. Yet PD's own exporter (`hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java`, lines 68-79) only publishes `hg.up`, `hg.graphs`, `hg.stores`, `hg.terms`, `hg.partition.terms{id}`, `hg.partitions{graph}`, and `hg.graph.size{graph}`. PD knows every store's state and every shard's role and does not export any of it either.

### Proposal

Export a small set of recovery-oriented families on the existing actuator endpoint, following the current naming style (dotted micrometer names under the `hg.` prefix with the `hg=store` common tag, low-cardinality tags like the existing `graph` and `group`):

Store side, registered next to the existing gauges in `StoreMetrics` / a new `RecoveryMetrics` source in `MetricsConfig`:

- `hg.partition.state{graph, partition, state}` gauge: 1 when the partition's work state equals `state` (`PState_Normal`, `PState_Warn`, `PState_Offline`, `PState_Fault`), else 0. Values come straight from `Partition.getWorkState()`, already read in `HgMetricService.getStoreMetrics`.
- `hg.partitions.recovering` gauge: count of local partitions whose work state is not `PState_Normal`. Gives dashboards a single "still recovering" number per store.
- `hg.raft.snapshot.installing{group}` gauge: 1 between `HgSnapshotHandler.onSnapshotLoad` start and finish.
- `hg.raft.snapshot.install.bytes{group}` and `hg.raft.snapshot.install.bytes.total{group}` gauges: bytes loaded so far versus total size of the snapshot data directory, both observable inside `onSnapshotLoad`.
- `hg.raft.log.lag{group}` gauge: leader committed index minus local applied index, i.e. the distance still to replay after snapshot install. The local side is `PartitionEngine.getCommittedIndex`; the leader side is already shipped to PD in `RaftStats`.
- `hg.raft.leader.count` gauge: size of `HgStoreEngine.getLeaderPartition()`. During and after recovery this shows leadership draining back to the rebuilt node.

PD side, complementary and cheap because the data is already in `StoreNodeService`:

- `hg.store.state{store, state}` gauge: per registered store, 1 for the current `StoreState` (`Up`, `Offline`, `Exiting`, `Tombstone`, ...).
- `hg.store.leader.count{store}` gauge: leaders per store from the shard group table, so balance and recovery are visible even while the recovering store itself cannot serve its endpoint.

All of these are plain gauges over state the process already holds, so the cost is a handful of map lookups per scrape, in line with how `StoreMetrics` and `PDMetrics` work today.

### Context

While testing distributed deployments on Kubernetes, I observed this during fault tests that kill a Store pod, drop its volume, and verify the cluster heals: grading "healed" today means watching readiness probes and grepping logs, which is exactly the gap described above. For reference, the deployment tooling used for these tests is the Helm chart contributed in PR #3132 (issue #3131); it already scrapes the standard actuator endpoints of Store and PD, so these families would light up recovery dashboards and alerts with no deployment changes at all. I am happy to work on a PR for the store-side gauges if the direction sounds right to the maintainers.

Contributor guide

Open the contributing guide

Research direction

Start with hg-store-node/.../metrics/MetricsConfig.java and StoreMetrics.java, then trace the recovery state and indexes through HgStoreStateMachine, HgSnapshotHandler, PartitionEngine, HgMetricService, and HgStoreEngine. Review the complementary PD path in PDMetrics.java and StoreNodeService.java. Done means the proposed store and PD recovery gauges are exposed on the existing Prometheus endpoints with the stated labels and values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, prometheus, spring-boot
Domain
backend, databases, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.