Webhooks get context deadline exceeded

Pulling this in from Slack for tracking - After some time I've been unable to reliably resolve this. I am running the VM Operator on 6+ cluster but only getting this frequently on a few clusters, and I could not identify and correlation as to why.

After upgrading to v0.47.3 (helm chart 0.34.7) using the helm chart, I'm getting these errors from server-side apply:

Internal error occurred: failed calling webhook "vmagent.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-operator.monitoring.svc/validate-operator-victoriametrics-com-v1beta1-vmagent?timeout=10s": context deadline exceeded

I tried disabling the vmagent webhook, and then I started to get the same error with other resources like vmalert. There are no errors in the logs that seem directly related.

I've been able to confirm that we can connect to the service, it just seems to not respond to the webhook posts.

Start with the VM Operator webhook service and the vmagent and vmalert admission requests described in the report. Reproduce the timeout with server-side apply on affected and unaffected clusters, then compare their webhook behavior and logs. Done means the intermittent response timeout has a verified cause and a documented fix or concrete diagnosis.