prometheus / prometheus/alertmanager
Feature Request: Support for Dynamic Merging or Patching of Routing Tree Subtrees
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
Currently, Alertmanager's routing tree is defined statically in the alertmanager.yml configuration file. While this approach works well for environments where configurations are managed centrally, it presents challenges in dynamic or multi-tenant environments, where multiple teams or systems need to share the same Alertmanager instance.
Problem Statement
Managing notifications through Alertmanager routing trees in dynamic or multi-tenant setups is difficult because:
- Lack of Modularity: The routing tree is treated as a single, static object. It is not possible to dynamically merge or patch a subtree into the existing routing tree without regenerating the entire configuration.
- Scalability: In shared Alertmanager instances, each team may have specific routing requirements. Currently, these must be handled in a centralized, monolithic configuration file, which can become unwieldy and error-prone as the number of teams or requirements grows.
- Lack of Dynamic Flexibility: Dynamic environments, such as Kubernetes clusters, often require updates to routing configurations based on events (e.g., new clusters or services). This is difficult to achieve without external tooling to regenerate and reload the configuration.
Use Case
In a shared Grafana + Alertmanager setup, different teams want to manage their own notification routing policies independently, while still using a central Alertmanager instance. For example:
- Team A wants to route all alerts with label
team="A"to their Slack channel. - Team B wants to route alerts with label
team="B"to their PagerDuty service.
Today, this would require a centralized admin to maintain and update the monolithic configuration file, or for each team to run its own instance of Alertmanager.
Proposed Solution
Introduce support for dynamically merging or patching subtrees into the routing tree. This could be achieved through:
- Dynamic Subtree Injection: Allow teams or systems to submit their subtree configuration (e.g., via an API or separate file) to be merged into the main routing tree at a specified location.
- Granular Reloading: Instead of requiring a full configuration reload, allow for partial reloads where only the modified subtree is reloaded.
- Modular Configuration: Support breaking the routing tree into modular files or objects that can be updated independently and then aggregated by Alertmanager.
I hope this makes sense to you. We encountered this problem all the way downstream in crossplane-provider-grafana. Here is also the related grafana issue.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Alertmanager's existing routing-tree configuration and reload behavior; no specific files or tests are named in the issue. Read the linked crossplane-provider-grafana and Grafana Grizzly issues for context, then define the intended merge or patch API, subtree placement, and reload semantics before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100