ci: wait-rollouts image guard compares against a stale image list
- Dominant language
- Go
- Stars
- 28
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 55
Description
Found while reviewing #619.
`hack/release/wait-rollouts.sh` resolves the workload's desired images **once**, before the wait begins:
```
meta="$(resolve_target "$target")" # wait_target, before wait_rollout
...
wait_rollout "$target" "$kind" "$selector" "$desired"
```
`check_images` then filters pod failures against that list, dropping any container whose image is not in it. The comment explains why the filter exists - it stops pods from a superseded revision gating the current rollout - and that reasoning is sound.
The problem is that component workloads are updated asynchronously by the operator, so the list can be captured *before* the update. If the operator then rewrites the template and the **new** image is the broken one, its ImagePullBackOff is filtered out as "not an image this workload wants", and the guard goes quiet on precisely the failure it exists to name. The run falls back to a bare `rollout status` timeout, which is the 15-consecutive-red-nights failure mode #611 set out to fix.
#619 hit the same staleness in the tolerance check and resolved it by reading the object on every poll:
> The tag is matched against the CURRENT object on every poll, never against the spec read once before the wait began, because observing the operator rewrite that template mid-wait is the entire mechanism.
The same fix applies here. `node_tolerance` already re-reads the object each poll for a DaemonSet, so for those workloads the images are available at no extra cost; the question is how to share that read cleanly with `check_images` without giving the image guard a per-poll `get` for workloads that do not otherwise need one.
Covered by the harness in `hack/release/wait_rollouts_test.go`: a `setNth("getjson-ds_gantry", ...)` that flips the template mid-wait, plus a pod failing on the new image, is the shape of the test.
Contributor guide
Research direction
Start with hack/release/wait-rollouts.sh, tracing resolve_target, wait_rollout, node_tolerance, and check_images to understand when workload data is read. Use the scenario in hack/release/wait_rollouts_test.go with setNth("getjson-ds_gantry", ...) and a pod failing on the new image. Done means the guard detects that failure after the template changes without adding unnecessary reads for workloads that do not need them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, shell
- Domain
- ci-cd, devops, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100