GoogleCloudPlatform / GoogleCloudPlatform/cloud-sql-proxy-operator

upgradeDefaultProxyOnStartup races against reconciler, produces consistent ERROR logs on every operator restart

Đang mở
#764 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Go
Star
120
Fork
18
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### Problem

Every time the operator pod starts, the logs reliably contain multiple ERROR lines like:

\`\`\`
ERROR unable to update AuthProxyWorkload during startup upgrade
{"error": "Operation cannot be fulfilled on authproxyworkloads.cloudsql.cloud.google.com
\"authproxyworkload-\": the object has been modified; please apply your changes
to the latest version and try again"}
github.com/GoogleCloudPlatform/cloud-sql-proxy-operator/internal/controller.(*upgradeDefaultProxyOnStartup).Start
/work/internal/controller/proxy_image_upgrade.go:58
\`\`\`

In our cluster (~40 AuthProxyWorkload resources across ~25 namespaces in prod), ≥10 of these errors fire on every operator restart.

### Root cause

\`upgradeDefaultProxyOnStartup.Start\` ([proxy_image_upgrade.go:38–62](https://github.com/GoogleCloudPlatform/cloud-sql-proxy-operator/blob/main/internal/controller/proxy_image_upgrade.go#L38-L62)) does:

1. \`List\` all \`AuthProxyWorkload\` resources (snapshot at resourceVersion T).
2. For each one with a default proxy image, call \`c.c.Update(ctx, &p)\` with the (now potentially stale) object.

But the \`AuthProxyWorkload\` reconciler is **already running** by the time \`Start\` is called (both are registered as runnables on the manager, and reconcile is event-driven so it fires as soon as the cache is populated). The reconciler typically updates the resource's status/annotations during its reconcile pass.

So the timeline is:
- T0: \`Start\` calls \`c.c.List\` → snapshot at resourceVersion V₀.
- T1: reconciler fires for \`apw1\` → updates apw1 → resourceVersion now V₁.
- T2: \`Start\` iterates to \`apw1\`, calls \`c.c.Update(ctx, &p)\` with resourceVersion V₀ → apiserver rejects (409, "the object has been modified").

The conflict is inevitable for any workload that the reconciler manages to touch between \`List\` and \`Update\`.

\`NeedLeaderElection()\` returning \`true\` means this happens on the single leader pod; replicas count doesn't matter — we observed the same pattern with \`replicas: 1\` and \`replicas: 2\`.

### Functional impact

The startup-upgrade fails silently for the conflicted workloads, but the regular reconcile loop subsequently catches up — workloads with default proxy images still get the new image via normal reconciliation. So **no behavioral problem**, just persistent log noise that pollutes alerting on operator restarts.

### Reproduction

Any cluster with multiple \`AuthProxyWorkload\` resources will reproduce this on every operator restart. Restart the operator pod and check logs.

### Proposed fixes (in increasing scope)

1. **Retry-on-conflict in the startup-upgrade goroutine.** Wrap the \`c.c.Update(ctx, &p)\` call in \`retry.RetryOnConflict\` from k8s.io/client-go/util/retry. On 409, re-fetch and re-apply. Cheap fix; zero behavior change.

2. **Add a flag to disable.** \`--upgrade-on-startup=true|false\` (default \`true\` for back-compat). Lets operators opt out if image-version drift on operator restart isn't a concern for their use case.

3. **Replace startup-upgrade with annotation-based trigger.** Instead of forcing an Update on every restart, write the operator's default image version to a ConfigMap/Lease/CRD-status field. The reconciler can compare and trigger a reconcile only when the image version changes. Eliminates the race entirely; only does work when there's actually work to do.

Option (1) is the smallest patch and would resolve the noise. Option (2) gives operators a knob. Option (3) is the cleanest long-term shape but is a bigger refactor.

### Environment

- Operator version: \`v1.7.5\` (build \`3db049b\`)
- Manager runtime: \`go1.26.0 linux/amd64\`
- Kubernetes: \`v1.35.3-gke.1389000\` (GKE)
- Manifest source: \`https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy-operator/v1.7.5/cloud-sql-proxy-operator.yaml\`

Happy to send a PR for option (1) if it'd be welcome.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start in internal/controller/proxy_image_upgrade.go at upgradeDefaultProxyOnStartup.Start, then inspect how the AuthProxyWorkload reconciler updates resources and how Kubernetes conflict retries are used elsewhere. Reproduce by restarting the operator with multiple AuthProxyWorkload resources. Done means startup upgrades no longer emit avoidable conflict errors and default-image workloads still receive the upgrade.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
go, kubernetes
Lĩnh vực
backend, infrastructure
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
68/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.