[RFC]: Standardize Prometheus Metrics with prometheus-cpp

## Changes proposed

### Summary

Use a shared metrics module based on `prometheus-cpp-core` in Transfer Engine, TENT, and Mooncake Store. The module will own metric registration and Prometheus text serialization, while each service's HTTP stack remains responsible for serving `/metrics`.

### Motivation

Mooncake currently has three metrics implementations:

- yalanting static and dynamic metrics;
- Store-specific hybrid metrics and manual serialization;
- hand-written Prometheus text for selected Master metrics.

This duplicates registration, label management, serialization, and summary logic. A maintained Prometheus client gives Mooncake one data model and one standards-compliant serializer.

### Related Work

- #2958 adds standard Prometheus metrics to Classic Transfer Engine but explicitly defers backend unification.
- #3158 and #3385 decouple Mooncake Store business decisions from metrics; they are complementary and do not select a shared metrics backend.
- This RFC focuses on one cross-component recording, collection, and Prometheus exposition architecture.

### Goals

- Replace the current metrics implementations and delete the Store hybrid metrics implementation.
- Use `prometheus-cpp-core` for the external Prometheus data model and text format.
- Avoid adding prometheus-cpp's HTTP, push, compression, curl, or zlib components.
- Keep metric recording suitable for Transfer Engine and Store hot paths.
- Use bounded labels and preserve one stable metric schema across components.

### Non-Goals

- Introducing OpenTelemetry or push-based metrics.
- Letting metric code create or own an HTTP server.
- Adding compatibility adapters for the retired metrics API.

### Design

Add a shared `mooncake::metrics` module that provides:

- one `prometheus::Registry` per service process;
- sharded hot-path counters and histograms exposed as `prometheus::Collectable`;
- direct prometheus-cpp metrics for non-hot-path instrumentation;
- Prometheus text serialization for an HTTP handler;
- compile-time no-op recording when metrics are excluded from a build.

Services will retain their HTTP endpoint but obtain its response body from the shared registry. prometheus-cpp's `Exposer` will not be linked.

All bounded label combinations, such as transfer kind, transport, stage, RPC, and client operation, must be registered during initialization. The hot path will retain direct references to cache-line-separated shards selected by enums or array indices. Recording updates only exact buckets, counts, and sums. Collection merges the shards and creates cumulative Prometheus histogram buckets. The hot path must not construct label maps or call `Family::Add()` per observation.

Truly dynamic series, such as tenant or segment metrics, may be added and removed only in their control-plane lifecycle paths. Their cardinality must be bounded operationally.

Prometheus counters will remain monotonic. Interval throughput and rates will be calculated by PromQL or from non-destructive snapshots; recording code will not reset counters. Existing SSD summaries that duplicate histograms will be removed, and percentiles will be calculated from histogram buckets.

### Performance

The following microbenchmark mirrors the metric updates currently made by each data-path event. The proposed implementation records into 64 cache-line-separated shards and aggregates them during collection. Values are nanoseconds per event, shown as current to proposed:

| Recording path | 1 thread | 4 threads | 16 threads |
| --- | ---: | ---: | ---: |
| Legacy TE successful task | 29 -> 21 | 29 -> 12 | 36 -> 3 |
| Store Master RPC | 221 -> 21 | 456 -> 12 | 703 -> 3 |
| Store client operation | 278 -> 26 | 496 -> 15 | 842 -> 4 |
| Store SSD read | 89 -> 52 | 93 -> 31 | 116 -> 8 |
| TENT successful request | 1,100 -> 108 | 807 -> 62 | 1,042 -> 16 |

Both implementations were built with GCC `-O3`; construction, warm-up, and worker threads were pinned to the same NUMA locality. The table reports rounded medians from three independent processes, each taking the median of seven alternating runs after warm-up. The test uses one shared series per metric, representing contention when many workers perform the same operation. It measures recording only, not business logic, timestamps, registry collection, or HTTP scraping. Multi-threaded values are aggregate throughput normalized per event, not single-call latency.

Numeric collection cost approximately 2.8 microseconds for Store RPC, 3.1 microseconds for SSD read, and 9.3 microseconds for the TENT metric set. Continuous collection, an intentionally unrealistic worst case, increased the 16-thread SSD and TENT results from 7.8 to 11.5 and 15.6 to 22.2 nanoseconds per event respectively. Prometheus object construction and text serialization were not included.

This proves the contention can be moved off the hot path, but the prototype is not the production layout. Padding every metric shard uses about 24 KiB per Store RPC series and 132 KiB for one TENT metric set. A production version must pack related metrics into one bounded shard set and ensure each collected histogram remains internally consistent while writers are active.

Before migration is accepted, representative Transfer Engine, TENT, Store client, Master, and HA workloads must compare:

1. metrics disabled;
2. the current implementation;
3. sharded recording with continuous scraping.

The comparison must cover throughput, CPU use, and tail latency at several thread counts.

### Migration

1. Add the shared registry and serializer using `prometheus-cpp-core` only.
2. Migrate Store Master and HA metrics and delete their serialization lists.
3. Migrate Store client metrics and delete `hybrid_metric.h`.
4. Migrate TENT with pre-registered label combinations.
5. Migrate legacy Transfer Engine and remove counter resets.
6. Remove the retired metrics code, links, tests, and documentation.
7. Run component and end-to-end performance validation.

Each step must leave the affected component buildable and testable; the old and new metrics backends will not coexist after a component is migrated.

### Risks

- Histogram contention can regress highly concurrent static recording paths.
- Sharded recording can consume excessive memory or expose inconsistent histogram fields if its shard layout and snapshot protocol are naive.
- Unbounded dynamic labels can increase memory use and Prometheus cardinality.
- Metric names, types, labels, or buckets can silently break dashboards.

### Alternatives

- **Retain the current metrics implementation:** rejected because it preserves duplicate label management and serialization paths.
- **Use prometheus-cpp's HTTP server:** rejected because metrics must not select the service HTTP stack.
- **Adopt OpenTelemetry now:** rejected because it expands the migration and is unnecessary for the current Prometheus-only requirement.

## Before submitting a new issue...

- [x] Searched for relevant issues and reviewed the project documentation.

Start by tracing the existing metrics implementations in Transfer Engine, TENT, and Mooncake Store, including the Store `hybrid_metric.h` path and each service's `/metrics` handler. Compare the migration steps and performance validation requirements; done means the shared prometheus-cpp-core registry and serializer replace the retired paths while affected components remain buildable, testable, and performance-validated.

Written by the indexing model from the issue text.