Can not terminate existing alert for removed alert checks

### Bug Description

On Prometheus, we got some bcache alerts due to bcache was not configured in cloud environment. Removing "bcache-tuning" was a solution to remove bache alert checks. But, we still see those alerts as Active in Prometheus. We need to be able to clear them permanently.


ubuntu@oshci03-nue2:~$ /usr/lib/nagios/plugins/check_bcache.py /etc/tune-bcache-config.yaml
-bash: /usr/lib/nagios/plugins/check_bcache.py: No such file or directory

### To Reproduce

With openstack deployed;
juju deploy traefik
juju deploy alertmanager
juju deploy prometheus
juju deploy grafana
juju deploy catalogue
juju deploy loki

```
$ juju status --relations -m microk8s-controller:cos
Model Controller Cloud/Region Version SLA Timestamp
cos microk8s-controller microk8s_cloud/localhost 3.6.4 unsupported 09:43:53Z

App Version Status Scale Charm Channel Rev Address Exposed Message
alertmanager 0.27.0 active 1 alertmanager-k8s latest/stable 156 10.152.183.248 no
catalogue active 1 catalogue-k8s latest/stable 81 10.152.183.169 no
grafana 9.5.3 active 1 grafana-k8s latest/stable 139 10.152.183.27 no
loki 2.9.6 active 1 loki-k8s latest/stable 187 10.152.183.135 no
prometheus 2.52.0 active 1 prometheus-k8s latest/stable 232 10.152.183.246 no
scrape-interval-config n/a active 1 prometheus-scrape-config-k8s latest/stable 58 10.152.183.141 no
traefik 2.11.0 active 1 traefik-k8s latest/stable 234 10.152.183.111 no Serving at 10.250.136.22

Unit Workload Agent Address Ports Message
alertmanager/0* active idle 10.1.35.138
catalogue/0* active idle 10.1.100.137
grafana/0* active idle 10.1.100.140
loki/0* active idle 10.1.151.72
prometheus/0* active idle 10.1.35.139
scrape-interval-config/0* active idle 10.1.151.70
traefik/0* active idle 10.1.100.139 Serving at 10.250.136.22

Offer Application Charm Rev Connected Endpoint Interface Role
alertmanager alertmanager alertmanager-k8s 156 0/0 karma-dashboard karma_dashboard provider
grafana grafana grafana-k8s 139 4/4 grafana-dashboard grafana_dashboard requirer
loki loki loki-k8s 187 3/3 logging loki_push_api provider
prometheus prometheus prometheus-k8s 232 3/3 metrics-endpoint prometheus_scrape requirer
receive-remote-write prometheus_remote_write provider
scrape-interval-config scrape-interval-config prometheus-scrape-config-k8s 58 2/2 configurable-scrape-jobs prometheus_scrape requirer

Integration provider Requirer Interface Type Message
alertmanager:alerting loki:alertmanager alertmanager_dispatch regular
alertmanager:alerting prometheus:alertmanager alertmanager_dispatch regular
alertmanager:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
alertmanager:grafana-source grafana:grafana-source grafana_datasource regular
alertmanager:replicas alertmanager:replicas alertmanager_replica peer
alertmanager:self-metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
catalogue:catalogue alertmanager:catalogue catalogue regular
catalogue:catalogue grafana:catalogue catalogue regular
catalogue:catalogue prometheus:catalogue catalogue regular
catalogue:replicas catalogue:replicas catalogue_replica peer
grafana:grafana grafana:grafana grafana_peers peer
grafana:metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
grafana:replicas grafana:replicas grafana_replicas peer
loki:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
loki:grafana-source grafana:grafana-source grafana_datasource regular
loki:metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
loki:replicas loki:replicas loki_replica peer
prometheus:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
prometheus:grafana-source grafana:grafana-source grafana_datasource regular
prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer
scrape-interval-config:metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
traefik:ingress alertmanager:ingress ingress regular
traefik:ingress catalogue:ingress ingress regular
traefik:ingress-per-unit loki:ingress ingress_per_unit regular
traefik:ingress-per-unit prometheus:ingress ingress_per_unit regular
traefik:metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
traefik:peers traefik:peers traefik_peers peer
traefik:traefik-route grafana:ingress traefik_route regular
```

Then remove bcache-tuning;

juju remove-application bcache-tuning

### Environment

Openstack Jammy/Yoga and and microk8s 1.28.15``

### Relevant log output

```shell
Alert manager logs do not contain error;

2025-04-22T12:41:32.063Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: rate(alertmanager_notifications_failed_total{integration=~".*"}[5m]) > 0
2025-04-22T12:41:32.076Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_notifications_failed.rule
2025-04-22T12:41:32.092Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: absent(up{})
2025-04-22T12:41:32.106Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_missing.rule
2025-04-22T12:41:32.124Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: alertmanager_config_last_reload_successful{} != 1
2025-04-22T12:41:32.138Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_configuration_reload_failure.rule
2025-04-22T12:41:32.154Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: up < 1
2025-04-22T12:41:32.168Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: absent(up)
2025-04-22T12:41:32.299Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log No relation: certificates
2025-04-22T12:41:32.515Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log No relation: certificates
2025-04-22T12:41:32.536Z [container-agent] 2025-04-22 12:41:32 DEBUG juju-log applying config changes
2025-04-22T12:41:33.993Z [container-agent] 2025-04-22 12:41:33 WARNING juju-log config reload via HTTP POST failed: Bad response
2025-04-22T12:41:34.012Z [container-agent] 2025-04-22 12:41:34 INFO juju-log Restarting service alertmanager
2025-04-22T12:41:35.162Z [container-agent] 2025-04-22 12:41:35 WARNING juju-log charm tracing buffer exceeds max history length (100 events)
2025-04-22T12:41:35.762Z [container-agent] 2025-04-22 12:41:35 WARNING juju-log cannot determine if reload succeeded
2025-04-22T12:41:35.896Z [container-agent] 2025-04-22 12:41:35 WARNING juju-log charm tracing buffer exceeds max history length (100 events)
```

### Additional context

_No response_

Start by reproducing the deployment described with Prometheus and Alertmanager, then remove bcache-tuning and compare the remaining alert state with the missing /usr/lib/nagios/plugins/check_bcache.py check. Trace how removed alert checks are represented and cleared; done means alerts from the removed check no longer remain active in Prometheus.

Written by the indexing model from the issue text.