AlertManagerConfig is SLO-scoped but writes tenant-wide config: clobbering, cascading delete, and cross-tenant deletion

Summary

AlertManagerConfig is named and owned per-SLO, but its payload is the entire tenant's Alertmanager configuration. Every AlertManagerConfig in a tenant is a writer to one shared document, and nothing in OSKO detects or mediates that.

This is a modelling mismatch rather than a single bug, and it produces three concrete failures. The third is a cascading outage triggered by a routine operation.

Where the mismatch lives

AlertManagerConfigSpec carries no routing scope at all — just a pointer to a Secret:

type AlertManagerConfigSpec struct {
	SecretRef SecretRef `json:"secretRef,omitempty"`
}

The SLO controller creates one of these per SLO, named <slo>-alerting, referencing a Secret named <slo>-alerting-config. The name implies SLO scope. The content is tenant-wide: the Secret's alertmanager.yaml is the complete config, routes and receivers included.

The tenant comes from the Datasource (ds.Spec.ConnectionDetails.TargetTenant), so every SLO pointing at the same Datasource shares one destination.

1. Writes clobber, last one wins

internal/controller/osko/alertmanagerconfig_controller.go:177

err = r.MimirClient.CreateAlertmanagerConfig(ctx, string(yamlData), nil)

Mimir's POST /api/v1/alerts replaces the tenant's configuration wholesale. It does not merge. With five SLOs enabling magicAlerting against one Datasource, whichever reconciles last defines routing for all five.

In practice this is only stable when every Secret holds byte-identical content — which is an invariant no code enforces, nothing validates, and nothing documents as load-bearing. A user who tailors one SLO's routing silently rewrites everyone else's.

2. Deleting one SLO wipes alerting for the whole tenant

internal/controller/osko/alertmanagerconfig_controller.go:219

func (r *AlertManagerConfigReconciler) deleteAlertmanagerConfigAPI() error {
	if err := r.MimirClient.DeleteAlermanagerConfig(context.Background()); err != nil {
		return err
	}
	return nil
}

No scope argument — this deletes the tenant's entire Alertmanager config. It is called from a single AlertManagerConfig's finalizer (:105).

So deleting one SLO with magic alerting enabled cascades to its owned AlertManagerConfig, whose finalizer removes routing for every other SLO in that tenant. The survivors keep reporting Ready=True — their CRs are untouched, and only the remote state is gone — so nothing surfaces the breakage until an alert silently fails to route.

This is the most serious of the three: deleting an SLO is a routine operation with no obvious blast radius.

3. The cached client can delete the wrong tenant

MimirClient is a field on the reconciler, reused across reconciles:

type AlertManagerConfigReconciler struct {
	client.Client
	Scheme      *runtime.Scheme
	Recorder    record.EventRecorder
	MimirClient *mimirclient.MimirClient
}

The create path rebuilds it unconditionally (:171). The delete path only builds it when it is nil (:85):

if r.MimirClient == nil {
    // ...build from this AMC's datasourceRef...
}
if r.MimirClient != nil {
    r.deleteAlertmanagerConfigAPI()
}

With two Datasources on different tenants, a reconcile for tenant A leaves a client pointed at A. A subsequent deletion of an AlertManagerConfig belonging to tenant B reuses that stale client and issues the delete against tenant A. Combined with issue 2, deleting one SLO in one tenant can wipe Alertmanager configuration in a different tenant.

Suggested directions

Not prescribing an answer, but the options seem to be:

1. Make the scope honest. Move AlertManagerConfig from SLO-scoped to Datasource/tenant-scoped — one per tenant by construction, not one per SLO. This matches what the API actually does and makes the collision structurally impossible. Requires deciding what happens to magicAlerting's per-SLO ownership.
2. Merge instead of replace. Have OSKO compose the tenant config from all AlertManagerConfigs targeting that tenant, so each contributes routes/receivers rather than overwriting. Keeps the per-SLO model, but OSKO becomes responsible for merge semantics and conflict resolution.
3. Detect and refuse. Keep today's behaviour but fail loudly: if more than one AlertManagerConfig resolves to the same tenant with differing content, mark them Ready=False and emit an event rather than racing. Smallest change, turns a silent problem into a visible one.

Independently of which is chosen, issues 2 and 3 are bugs worth fixing on their own: the finalizer should not delete tenant-wide state on behalf of one SLO, and the delete path should not reuse a client built for a different tenant.

Notes

• Behaviour confirmed against Mimir 3.2.1.
• Current mitigation is documentation only — see docs/alertmanager-configs.md (added in #153), which tells users to keep every Secret in a tenant identical. That is a workaround for the modelling problem, not a fix.

Start with internal/controller/osko/alertmanagerconfig_controller.go, especially the reconciler, finalizer, create path around lines 171-177, and delete path around lines 85 and 219. Reproduce the shared-tenant overwrite and stale-client deletion cases, then determine which scope, merge, or refusal direction is selected; done should include tests covering tenant isolation and safe deletion, with docs/alertmanager-configs.md updated if behavior changes.