prometheus / prometheus/alertmanager

Add dynamic duration variable for the repeat_interval

Open
#3,186 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/config stale
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

  • System information:

    insert output of uname -srm here

  • Alertmanager version:

    insert output of alertmanager --version here (repeat for each alertmanager
    version in your cluster, if relevant to the issue)

  • Prometheus version:

    insert output of prometheus --version here (repeat for each prometheus
    version in your cluster, if relevant to the issue)

  • Alertmanager configuration file:

insert configuration here
  • Prometheus configuration file:
insert configuration here (if relevant to the issue)
  • Logs:
insert Prometheus and Alertmanager logs relevant to the issue here

For managing the repeat interval of the same alert, there is a configuration repeat_interval in the alertmanager below
repeat_interval : <duration>
The <duration> only supports for the pattern that matches regular expression below.
((([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?|0)

In my case, I want to set different repeat_interval for each prometheusrule in my k8s cluster.
Now, I can set "route" to change repeat_interval, but it could make the route configuration so dirty.
Example)
If I want to set 4 different repeat_intervals that depend on the alert severity, then there are 4 additional lines for every route config.
Let assume that there are 4 (severity, repeat_interval) pairs. (warning, 1h) (major, 30m) (critical, 10m) and (emergency, 1m)
Then the route should be...

route:
-  receiver: receiver1
   matchers:
   - severity="warning"
     repeat_interval: 1h
-  receiver: receiver1
   matchers:
   - severity="major"
     repeat_interval: 30m
-  receiver: receiver1
   matchers:
   - severity="critical"
     repeat_interval: 10m
-  receiver: receiver1
   matchers:
   - severity="emergency"
     repeat_interval: 1m

If there are several receivers, then the configuration would be 1000+ lines.

If there could be
repeat_inverval: {{ $labels.repeat }} instead of
repeat_interval: static string,
the configuration would be shorter and people could use repeat_interval much more flexible.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Alertmanager configuration and routing behavior around the repeat_interval field, including how alert labels are available to configuration values. Trace the route examples in the issue and determine the supported scope and validation needed for a dynamic duration. Done means repeat intervals can vary by alert as requested without requiring duplicated route entries, with coverage for the documented examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.