Restart pods in Kyma-managed namespaces when ClusterTrustBundle CA changes [EPIC]

Summary

The RT Bootstrapper needs to automatically restart pods in Kyma-managed namespaces (e.g., kyma-system, istio-system) whenever the CA certificate within a ClusterTrustBundle (CTB) has changed. This ensures that Kyma-owned workloads consuming the CTB pick up the updated CA without manual intervention.

Customer workloads are explicitly out of scope — only workloads within Kyma-managed namespaces are considered for automatic restarts. The feature is opt-in: customers (i.e., Kyma component owners) can enable it by setting the annotation accordingly.

This issue was identified as part of the incident tracked in kyma/backlog#9692.

Background

The RT Bootstrapper already mounts a ClusterTrustBundle as a projected volume via the webhook, opt-in via the annotation rt-cfg.kyma-project.io/add-cluster-trust-bundle: "true" (see Pod Manipulations docs).

When the CA bundle is rotated (e.g., bi-weekly in NS2/Sovereign Cloud environments), running pods are not automatically restarted. Workloads may therefore continue using a stale/cached CA, leading to SSL validation failures against the BTP backend.

Required Changes

1. Annotation opt-in for restart behavior

Extend the existing rt-cfg.kyma-project.io/add-cluster-trust-bundle annotation to support a new value that signals the RT Bootstrapper to restart the pod/workload when the CTB CA changes:

Note: The exact value (e.g., "restart-on-change") is a suggestion — the implementing team should agree on the final naming.

• If the annotation is absent or set to "true", no restart is triggered. This preserves full backward compatibility.
• Only workloads explicitly opting in via the new value will be restarted.

2. CA change detection and pod restart trigger

The RT Bootstrapper must watch for changes to ClusterTrustBundle resources and compare the current CA with the previously observed CA. When a change is detected:

• Identify all workloads in Kyma-managed namespaces (e.g., kyma-system, istio-system) carrying the opt-in annotation value.
• Trigger a rolling restart of the affected workloads (e.g., by patching the restartedAt annotation on the owning Deployment, DaemonSet, or StatefulSet).
• Never restart workloads in customer-owned namespaces, even if they carry the annotation.

3. RBAC — Permission to restart workloads in Kyma-managed namespaces

The RT Bootstrapper's service account must be granted the necessary RBAC permissions to restart workloads (i.e., patch deployments/daemonsets/statefulsets) within Kyma-managed namespaces. Permissions are intentionally scoped to these namespaces only — no cluster-wide patch permissions are required or granted.

Acceptance Criteria

• The annotation rt-cfg.kyma-project.io/add-cluster-trust-bundle accepts a new value (e.g., "restart-on-change") in addition to the existing "true".
• When set to "restart-on-change", the RT Bootstrapper triggers a rolling restart of the owning workload (Deployment / DaemonSet / StatefulSet) when the CTB CA changes.
• When set to "true" or absent, behavior is unchanged — no restart is triggered.
• RT Bootstrapper has RBAC permissions to restart workloads in Kyma-managed namespaces (e.g., kyma-system, istio-system).
• RT Bootstrapper detects CA changes in ClusterTrustBundle resources.
• Restarts are limited to workloads within Kyma-managed namespaces. Workloads in customer-owned namespaces are never restarted, even if annotated.
• Pod Manipulations documentation is updated to reflect the new annotation value and its semantics.
• Restart behavior is covered by tests.
• Clean logs, see https://github.com/kyma-project/rt-bootstrapper/issues/178#issuecomment-5538990937

Related

• kyma/backlog#9692 — Incident: NS2/Sovereign Cloud CA bundle rotation
• Pod Manipulations docs