Add generation-change predicates to controller watches

Summary

Zero predicate usage exists across all controller watch configurations. Every metadata-only change (label additions, annotation updates, resource version bumps) triggers a full reconciliation cycle. This is particularly wasteful for MCPServer reconciliation which performs expensive operations like image validation and StatefulSet comparison.

Severity: SHOULD FIX
Area: Controller Efficiency
Breaking: No

Location

• All controller SetupWithManager methods across cmd/thv-operator/controllers/

Problem

Example from a typical controller:

func (r *MCPServerReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&v1alpha1.MCPServer{}).
        Owns(&appsv1.StatefulSet{}).
        Complete(r)
}

No builder.WithPredicates(...) is applied to any watch.

Impact

• Metadata-only changes (e.g., adding a label) trigger full reconciliation
• MCPServer reconciliation is particularly expensive due to image validation, StatefulSet comparison, and Service reconciliation
• Higher API server load from unnecessary reconciliation cycles
• Increased log noise from no-op reconciliations
• In clusters with frequent metadata updates (e.g., from GitOps tools adding annotations), this causes continuous unnecessary work

Recommended Fix

Add generation-change predicates on primary resource watches:

import "sigs.k8s.io/controller-runtime/pkg/predicate"

func (r *MCPServerReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&v1alpha1.MCPServer{},
            builder.WithPredicates(predicate.GenerationChangedPredicate{})).
        Owns(&appsv1.StatefulSet{}).
        Complete(r)
}

Apply to all controller primary resource watches. Do NOT apply to Owns() watches since owned resource status changes need to trigger reconciliation.

Related Issues

• Remove redundant annotation-based reconcile triggers in config controllers (#4623)

Start in cmd/thv-operator/controllers/ and inspect each SetupWithManager method, distinguishing primary For watches from Owns watches. Add generation-change predicates only to the primary watches, then run the existing controller test suite and verify owned-resource status changes still trigger reconciliation.