prometheus / prometheus/alertmanager

Feature Request: Support for Dynamic Merging or Patching of Routing Tree Subtrees

Open
#4,151 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

Currently, Alertmanager's routing tree is defined statically in the alertmanager.yml configuration file. While this approach works well for environments where configurations are managed centrally, it presents challenges in dynamic or multi-tenant environments, where multiple teams or systems need to share the same Alertmanager instance.

Problem Statement

Managing notifications through Alertmanager routing trees in dynamic or multi-tenant setups is difficult because:

  1. Lack of Modularity: The routing tree is treated as a single, static object. It is not possible to dynamically merge or patch a subtree into the existing routing tree without regenerating the entire configuration.
  2. Scalability: In shared Alertmanager instances, each team may have specific routing requirements. Currently, these must be handled in a centralized, monolithic configuration file, which can become unwieldy and error-prone as the number of teams or requirements grows.
  3. Lack of Dynamic Flexibility: Dynamic environments, such as Kubernetes clusters, often require updates to routing configurations based on events (e.g., new clusters or services). This is difficult to achieve without external tooling to regenerate and reload the configuration.

Use Case

In a shared Grafana + Alertmanager setup, different teams want to manage their own notification routing policies independently, while still using a central Alertmanager instance. For example:

  • Team A wants to route all alerts with label team="A" to their Slack channel.
  • Team B wants to route alerts with label team="B" to their PagerDuty service.

Today, this would require a centralized admin to maintain and update the monolithic configuration file, or for each team to run its own instance of Alertmanager.

Proposed Solution

Introduce support for dynamically merging or patching subtrees into the routing tree. This could be achieved through:

  1. Dynamic Subtree Injection: Allow teams or systems to submit their subtree configuration (e.g., via an API or separate file) to be merged into the main routing tree at a specified location.
  2. Granular Reloading: Instead of requiring a full configuration reload, allow for partial reloads where only the modified subtree is reloaded.
  3. Modular Configuration: Support breaking the routing tree into modular files or objects that can be updated independently and then aggregated by Alertmanager.

I hope this makes sense to you. We encountered this problem all the way downstream in crossplane-provider-grafana. Here is also the related grafana issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Alertmanager's existing routing-tree configuration and reload behavior; no specific files or tests are named in the issue. Read the linked crossplane-provider-grafana and Grafana Grizzly issues for context, then define the intended merge or patch API, subtree placement, and reload semantics before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.