prometheus / prometheus/alertmanager

amtool silence add shows unexpected behaviour interpreting matchers

Open
#3,385 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/cli
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?

amtool silence add shows unexpected behaviour interpreting matchers.

The --help for silence add shows that multiple matchers can be passed when creating a silence, separated via whitespace:

usage: amtool silence add [<flags>] [<matcher-groups>...]

Add a new alertmanager silence

    Amtool uses a simplified Prometheus syntax to represent silences. The
    non-option section of arguments constructs a list of "Matcher Groups"
    that will be used to create a number of silences. The following examples
    will attempt to show this behaviour in action:

    amtool silence add alertname=foo node=bar

  	This statement will add a silence that matches alerts with the
  	alertname=foo and node=bar label value pairs set.

However, if alertname is omitted and the first matcher in the matcher group does not contain an = or =~ then it will be assumed to be the value of the alertname. For example:

    amtool silence add foo node=bar

  	If alertname is omitted and the first argument does not contain a '=' or a
  	'=~' then it will be assumed to be the value of the alertname pair.

The problem here is that when this happens, sometimes a valid Prometheus-like matcher can be interpreted as the value for the alertname. For example:

// creates a silence for alertname="{foo=bar,bar=baz}"
amtool silence add '{foo=bar,bar=baz}' --comment="This is a comment"

amtool silence
9393d2dc-0eac-4e6b-8348-3711d96bc737  alertname="{foo=bar,bar=baz}"  2023-06-06 11:41:54 UTC  grobinson   This is a comment

Another example is the following:

// creates a silence for foo="bar,bar=baz"
amtool silence add 'foo=bar,bar=baz' --comment="This is a comment"

amtool silence
3cc077a0-6777-47d2-ada9-565c34a70e58  foo="bar,bar=baz"              2023-06-06 11:41:23 UTC  grobinson   This is a comment

What did you expect to see?

There are two issues here:

  1. It seems to me that the heuristic for detecting if alertname should be added is incorrect when passing a PromQL-like matcher such as {foo=bar,bar=baz}
  2. Both PromQL-like matchers such as {foo=bar,bar=baz} and comma separated matchers such as foo=bar,bar=baz should either be parsed or rejected, not coerced into something else

Environment

  • Alertmanager version:
alertmanager, version 0.25.0 (branch: main, revision: 83304da5070d8a12642b7a027d4c59060813c0e3)
  build user:       grobinson@Georges-MacBook-Air.fritz.box
  build date:       20230606-10:24:03
  go version:       go1.20.4
  platform:         darwin/arm64
  tags:             unknown

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the amtool silence add command and reproduce the documented matcher-group examples, including {foo=bar,bar=baz} and foo=bar,bar=baz. Trace how the first argument is classified as an alertname, then determine whether PromQL-like and comma-separated matcher forms should be parsed or rejected; done means neither form is silently coerced into an unintended matcher.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.