argoproj / argoproj/argo-workflows
Regression in v4.0: controller-side image entrypoint lookup couples to runtime.GOARCH (introduced by #15059)
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`.
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
**Summary**
#15059 (closing #15058) added `WithPlatform(runtime.GOARCH)` to the controller's image manifest lookup, defaulting to the controller's own architecture. This fixed #15058's case (arm-only image + arm controller defaulting to amd64) but introduced the inverse regression: any single-arch user image whose architecture doesn't match the controller's GOARCH now fails introspection with:
```
emissary: no child with platform linux/ in index
```
**Affected scenarios:**
| Controller arch | User image | v3.7 | v4.0+ |
|---|---|---|---|
| arm64 | amd64-only manifest list | ✅ | ❌ |
| amd64 | arm64-only manifest list | ✅ | ❌ |
| any | multi-arch | ✅ | ✅ |
| any | single OCI manifest (not a list) | ✅ | ✅ |
**Real-world impact**
We hit this running argo-workflows controllers on AWS Graviton (arm64) nodes for cost while in-house images are amd64-only (GitHub-hosted CI defaults to `linux/amd64`). Every internal workflow image failed introspection after upgrading v3.7.6 → v4.0.5. Same shape as the reproduction below.
The workflow pod itself would run fine — `nodeSelector` routes it to the matching-arch node. The break is purely controller-side: the controller can't read the entrypoint from the registry because it's filtering for its own architecture in the manifest list.
**Expected behavior**
The controller should successfully introspect any image whose manifest list contains at least one platform variant. The OCI `ENTRYPOINT`/`CMD` are defined in the Dockerfile and are identical across arch variants in virtually all real-world images — the controller doesn't actually need the matching-platform variant to read them, it just needs any config readable from the manifest.
**Workarounds**
1. **Pin the controller to match user image arch** (we ended up doing this — moved our controller to amd64 nodes).
2. Make every user image multi-arch. Costly to roll out across an entire org just for workflow controller compatibility.
### Version(s)
v4.0.0 – v4.0.5 (confirmed on v4.0.5). v3.7.x not affected (`remote.Image()` was called without `WithPlatform`).
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: arch-mismatch-demo-
spec:
entrypoint: main
templates:
- name: main
container:
image: arm64v8/alpine:latest # public arm64-only manifest list
```
Submit on a controller running on amd64. Fails immediately at controller-side introspection. Submit the same workflow with any amd64-only image on an arm64 controller — fails with the opposite error (`no child with platform linux/arm64`).
### Logs from the workflow controller
```
failed to look-up entrypoint/cmd for image "arm64v8/alpine:latest":
emissary: no child with platform linux/amd64 in index arm64v8/alpine:latest
```
### Logs from in your workflow's wait container
```
No wait container — pod never created. Failure happens during controller-side
manifest introspection, before pod spec generation.
```
### Source
The change introducing this behavior:
https://github.com/argoproj/argo-workflows/blob/v4.0.5/workflow/controller/entrypoint/container_registry_index.go#L32-L36
```go
var defaultPlatform = pkgv1.Platform{
Architecture: runtime.GOARCH, // ← couples to controller's arch
OS: runtime.GOOS,
}
img, err := remote.Image(ref, remote.WithAuthFromKeychain(kc), remote.WithPlatform(defaultPlatform))
```
In v3.7.6, the call was platform-agnostic:
```go
img, err := remote.Image(ref, remote.WithAuthFromKeychain(kc))
```
### Proposed fix
**Smallest possible (Option A - restore v3 fallback)**
```go
img, err := remote.Image(ref, remote.WithAuthFromKeychain(kc), remote.WithPlatform(defaultPlatform))
if err != nil && strings.Contains(err.Error(), "no child with platform") {
// Image's manifest list doesn't contain controller's arch.
// ENTRYPOINT/CMD are virtually always identical across arch variants,
// so any available variant suffices for emissary's purposes.
img, err = remote.Image(ref, remote.WithAuthFromKeychain(kc))
}
```
Preserves the #15058 fix (controller's preferred arch tried first) while restoring v3 behavior as a graceful fallback. ~5 lines + a unit test.
**Configurable default (Option B - operator escape hatch)**
Add a `defaultImagePlatform` field to `workflow-controller-configmap`:
```yaml
defaultImagePlatform: linux/amd64 # explicit; falls back to runtime.GOARCH if unset
```
Backward compatible (current runtime.GOARCH behavior remains the default when unset).
**Best (combination)**
1. defaultImagePlatform from controller config if set (Option B — operator default)
2. Else runtime.GOARCH (current behavior — preserves #15059's fix)
3. On no child with platform error: retry without platform filter (Option A — graceful fallback)
Happy to send the PR if maintainers prefer.
Contributor guide
Research direction
Start with workflow/controller/entrypoint/container_registry_index.go and inspect the existing controller/entrypoint tests. Reproduce the mismatch with the provided arm64v8/alpine image or an equivalent single-architecture image, then verify that lookup retries without the platform filter when the selected architecture is absent while preserving the preferred-platform behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100