prometheus / prometheus/alertmanager

API v3alpha: build stored-mutation watch infrastructure

Open
#5,480 1 comment 0 reactions 1 assignee View on GitHub

@siavashs is already working on this.

Since Aug 29, 2026.

component/api component/storage kind/enhancement
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

Parent: #5450

Summary

Build a transport-neutral, in-process stored-mutation substrate for AlertService and SilenceService watch RPCs.

The broker must be bounded and non-blocking so a slow client cannot stall alert ingestion, dispatch, silence storage, HA merge, configuration reload, or other API traffic.

Mutation broker

Provide:

  • Atomic snapshot-plus-subscribe handoff.
  • An explicit snapshot-complete event.
  • Monotonic node-local sequence numbers.
  • Immutable before/after payloads and resource IDs.
  • Bounded, non-blocking per-subscriber queues.
  • Explicit ResourceExhausted and resnapshot behavior on overflow.
  • Cancellation cleanup, stream count, lifetime, and idle controls.
  • Metrics for subscriptions, active consumers, overflow, and forced closure.

Cursors are node-local and non-durable. They do not promise replay after restart, failover, or reconnection to another HA peer.

Domain mutation feeds

Alerts

Extend the memory alert provider with non-blocking publication for accepted create, update, delete, and GC mutations. Do not attach one existing provider subscription per RPC client: current listeners can block alert ingestion while the provider mutex is held.

Alert groups

Add dispatcher/group and group-marker observers for stored group creation, update, deletion, content, and marker mutations. Publish only after releasing hot-path locks.

Do not synthesize mutations solely because silence/inhibition evaluation, route configuration, wall-clock time, or time intervals changed computed output.

Silences

Publish local Set/Expire mutations, replacement expire-plus-create, accepted HA Merge changes, and GC purge. Natural pending-to-active and active-to-expired transitions are derived from timestamps and are not mutation events.

Filtering

Derive entered, updated, and exited transitions from immutable before/after data as resources cross subscription filters. Avoid unbounded per-client membership state.

Acceptance criteria

  • Snapshot and subscription handoff cannot lose an accepted mutation.
  • Every snapshot ends with an explicit snapshot-complete marker.
  • Slow subscribers cannot block domain hot paths.
  • Queue overflow terminates with documented resnapshot semantics.
  • Cancellation and forced closure release queues and admission capacity.
  • Node-local cursor scope and restart/HA limitations are documented.
  • Alert, group, marker, local silence, HA silence, replacement, and GC mutations are covered.
  • Timer-only and recomputed transitions are not emitted.
  • Ginkgo/Gomega race and backpressure specs cover ordering, overflow, cancellation, HA merge, and lock isolation.

Dependencies

  • Shared contracts and filter semantics: #5479.
  • Stream admission, lifetime, and observability policy: #5478.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.