Expired silences are not garbage collected after -data.retention has been changed from 2400h to 169h.

What did you do?

Initially, I set -data.retention to 2400h.
Then I found that there are too many (close to 5000) expired silences (this make API and UI response slow).
I wanted to gc these expired silences, so I changed -data.retention from 2400h to 169h and restarted.
But the reboot didn't work, I waited for days, and the expired silences were still close to 5000

What did you expect to see?

Manually or automatically gc expired silences

What did you see instead? Under which circumstances?

But the reboot didn't work, I waited for days, and the expired silences were still close to 5000

Environment

• System information:

  Linux 3.10.0-1160.59.1.el7.x86_64 x86_64

• Alertmanager version:

alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)
build user: root@265f14f5c6fc
build date: 20220325-09:31:33
go version: go1.17.8
platform: linux/amd64

• Prometheus version:

  v2.39.1

• Alertmanager configuration file:

global:
  resolve_timeout: 99y
  ...
route:
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 1w

• Alertmanager CLI: (installed with Helm Chart)

alertmanager --storage.path=/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --cluster.advertise-address=[10.244.14.187]:9094 --cluster.listen-address=0.0.0.0:9094 --cluster.peer=monitor-alertmanager-0.monitor-alertmanager-headless:9094 --cluster.peer=monitor-alertmanager-1.monitor-alertmanager-headless:9094 --cluster.peer=monitor-alertmanager-2.monitor-alertmanager-headless:9094 --data.retention=169h --web.route-prefix=/ --log.level=info

• Prometheus configuration file:

insert configuration here (if relevant to the issue)

• Logs: (Rollout daemonset after modifying the configuration)

ts=2023-05-30T04:40:11.105Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=HEAD, revision=f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)"
ts=2023-05-30T04:40:11.105Z caller=main.go:232 level=info build_context="(go=go1.17.8, user=root@265f14f5c6fc, date=20220325-09:31:33)"
ts=2023-05-30T04:40:14.455Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
ts=2023-05-30T04:40:14.497Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
ts=2023-05-30T04:40:14.524Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml
ts=2023-05-30T04:40:14.532Z caller=main.go:535 level=info msg=Listening address=:9093
ts=2023-05-30T04:40:14.532Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
ts=2023-05-30T04:40:16.455Z caller=cluster.go:705 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=3 elapsed=2.00026447s
ts=2023-05-30T04:40:24.458Z caller=cluster.go:697 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003305543s
ts=2023-05-30T04:40:29.469Z caller=cluster.go:470 level=warn component=cluster msg=refresh result=failure addr=monitor-alertmanager-1.monitor-alertmanager-headless:9094 err="1 error occurred:\n\t* Failed to resolve monitor-alertmanager-1.monitor-alertmanager-headless:9094: lookup monitor-alertmanager-1.monitor-alertmanager-headless on 10.96.0.10:53: no such host\n\n"

Finally, my only expectation is to gc all the expired silences as soon as possible, even if I do it manually. Since this is a production environment, another expectation is that there will be no interruptions during the repair operation.

Thanks!

Start by tracing Alertmanager's silence storage and garbage-collection entry points around the --data.retention flag, then reproduce the behavior by changing retention from 2400h to 169h and restarting. Done means expired silences are removed promptly after retention changes, with no service interruption and a regression test covering the scenario.