VirtualMCPServer reconciler hot-loops when external controller annotates pod template

Summary

The VirtualMCPServer controller enters a continuous reconcile loop on clusters where any admission webhook, service mesh sidecar injector (Istio, Linkerd), or cloud controller (GKE node-pool annotations, cert-manager CA injector) writes labels or annotations to the Deployment's pod template. Each reconcile triggers a Deployment update, which increments metadata.generation, which re-enqueues the reconcile, without bound. On plain clusters with no external annotators the bug is latent and produces no visible symptom.

Symptom

• Continuous "Updating Deployment" log lines from ensureDeployment
• Steady stream of DeploymentUpdated events on the VirtualMCPServer object
• Elevated controller_runtime_reconcile_total counter with no backoff
• Potential resourceVersion conflicts logged when a concurrent writer races the update

Root cause

podTemplateMetadataNeedsUpdate in virtualmcpserver_controller.go uses maps.Equal for both labels and annotations:

if !maps.Equal(deployment.Spec.Template.Labels, expectedPodTemplateLabels) {
    return true
}
if !maps.Equal(deployment.Spec.Template.Annotations, expectedPodTemplateAnnotations) {
    return true
}

maps.Equal requires exact equality. If an external controller adds a single key to the pod template that buildPodTemplateMetadata does not produce — which only manages labelsForVirtualMCPServer labels and one checksum annotation — the check returns true and an update is triggered.

The write path in ensureDeployment then does:

deployment.Spec.Template = newDeployment.Spec.Template

This wholesale replacement strips every label and annotation from the live pod template that the operator did not itself produce. The external controller re-adds its key on the next pod admission or watch reconcile, podTemplateMetadataNeedsUpdate sees drift again, and the cycle continues.

Note that the Deployment's own (non-pod-template) metadata is not affected: deploymentMetadataNeedsUpdate already uses a key-iteration subset check and is correct.

Precedent

PR #5731 fixed the identical pattern for Service metadata: replaced maps.Equal with ctrlutil.MapIsSubset in needs-update checks, and replaced wholesale label/annotation assignment with ctrlutil.MergeLabels/ctrlutil.MergeAnnotations on the write path. That fix was not applied to the Deployment pod template path.

Proposed fix

Part 1 — detection, replace maps.Equal with ctrlutil.MapIsSubset:

if !ctrlutil.MapIsSubset(expectedPodTemplateLabels, deployment.Spec.Template.Labels) {
    return true
}
if !ctrlutil.MapIsSubset(expectedPodTemplateAnnotations, deployment.Spec.Template.Annotations) {
    return true
}

Part 2 — write path, preserve existing pod template metadata before the wholesale replacement, then merge:

existingPodLabels      := deployment.Spec.Template.Labels
existingPodAnnotations := deployment.Spec.Template.Annotations
deployment.Spec.Template = newDeployment.Spec.Template
deployment.Spec.Template.Labels      = ctrlutil.MergeLabels(newDeployment.Spec.Template.Labels, existingPodLabels)
deployment.Spec.Template.Annotations = ctrlutil.MergeAnnotations(newDeployment.Spec.Template.Annotations, existingPodAnnotations)

MergeLabels/MergeAnnotations give operator-owned keys precedence on collision and preserve external-only keys — identical to the Service fix in #5731.

Impact

The bug is latent on plain clusters (no admission webhooks that touch pod templates) and fires immediately on any cluster running Istio, Linkerd, cert-manager CA injector, GKE Workload Identity, or any custom admission webhook that mutates pod template metadata.

Related

See also: companion bugs for imagePullRefsHashAnnotation (#5817) and podTemplateSpecHashAnnotation (#5818), which are the same class of hot-reconcile-loop bug but driven by a different mechanism (stranded deployment-level annotations rather than external-annotation drift).