argoproj / argoproj/argo-workflows

Regression in v4.0: controller-side image entrypoint lookup couples to runtime.GOARCH (introduced by #15059)

Open
#16,258 2 comments 7 reactions 0 assignees View on GitHub
type/regression
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`.
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

**Summary**

#15059 (closing #15058) added `WithPlatform(runtime.GOARCH)` to the controller's image manifest lookup, defaulting to the controller's own architecture. This fixed #15058's case (arm-only image + arm controller defaulting to amd64) but introduced the inverse regression: any single-arch user image whose architecture doesn't match the controller's GOARCH now fails introspection with:

```
emissary: no child with platform linux/ in index
```

**Affected scenarios:**

| Controller arch | User image | v3.7 | v4.0+ |
|---|---|---|---|
| arm64 | amd64-only manifest list | ✅ | ❌ |
| amd64 | arm64-only manifest list | ✅ | ❌ |
| any | multi-arch | ✅ | ✅ |
| any | single OCI manifest (not a list) | ✅ | ✅ |

**Real-world impact**

We hit this running argo-workflows controllers on AWS Graviton (arm64) nodes for cost while in-house images are amd64-only (GitHub-hosted CI defaults to `linux/amd64`). Every internal workflow image failed introspection after upgrading v3.7.6 → v4.0.5. Same shape as the reproduction below.

The workflow pod itself would run fine — `nodeSelector` routes it to the matching-arch node. The break is purely controller-side: the controller can't read the entrypoint from the registry because it's filtering for its own architecture in the manifest list.

**Expected behavior**

The controller should successfully introspect any image whose manifest list contains at least one platform variant. The OCI `ENTRYPOINT`/`CMD` are defined in the Dockerfile and are identical across arch variants in virtually all real-world images — the controller doesn't actually need the matching-platform variant to read them, it just needs any config readable from the manifest.

**Workarounds**

1. **Pin the controller to match user image arch** (we ended up doing this — moved our controller to amd64 nodes).
2. Make every user image multi-arch. Costly to roll out across an entire org just for workflow controller compatibility.

### Version(s)

v4.0.0 – v4.0.5 (confirmed on v4.0.5). v3.7.x not affected (`remote.Image()` was called without `WithPlatform`).

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: arch-mismatch-demo-
spec:
entrypoint: main
templates:
- name: main
container:
image: arm64v8/alpine:latest # public arm64-only manifest list
```

Submit on a controller running on amd64. Fails immediately at controller-side introspection. Submit the same workflow with any amd64-only image on an arm64 controller — fails with the opposite error (`no child with platform linux/arm64`).

### Logs from the workflow controller

```
failed to look-up entrypoint/cmd for image "arm64v8/alpine:latest":
emissary: no child with platform linux/amd64 in index arm64v8/alpine:latest
```

### Logs from in your workflow's wait container

```
No wait container — pod never created. Failure happens during controller-side
manifest introspection, before pod spec generation.
```

### Source

The change introducing this behavior:

https://github.com/argoproj/argo-workflows/blob/v4.0.5/workflow/controller/entrypoint/container_registry_index.go#L32-L36

```go
var defaultPlatform = pkgv1.Platform{
Architecture: runtime.GOARCH, // ← couples to controller's arch
OS: runtime.GOOS,
}
img, err := remote.Image(ref, remote.WithAuthFromKeychain(kc), remote.WithPlatform(defaultPlatform))
```

In v3.7.6, the call was platform-agnostic:

```go
img, err := remote.Image(ref, remote.WithAuthFromKeychain(kc))
```

### Proposed fix

**Smallest possible (Option A - restore v3 fallback)**

```go
img, err := remote.Image(ref, remote.WithAuthFromKeychain(kc), remote.WithPlatform(defaultPlatform))
if err != nil && strings.Contains(err.Error(), "no child with platform") {
// Image's manifest list doesn't contain controller's arch.
// ENTRYPOINT/CMD are virtually always identical across arch variants,
// so any available variant suffices for emissary's purposes.
img, err = remote.Image(ref, remote.WithAuthFromKeychain(kc))
}
```

Preserves the #15058 fix (controller's preferred arch tried first) while restoring v3 behavior as a graceful fallback. ~5 lines + a unit test.

**Configurable default (Option B - operator escape hatch)**

Add a `defaultImagePlatform` field to `workflow-controller-configmap`:

```yaml
defaultImagePlatform: linux/amd64 # explicit; falls back to runtime.GOARCH if unset
```

Backward compatible (current runtime.GOARCH behavior remains the default when unset).

**Best (combination)**

1. defaultImagePlatform from controller config if set (Option B — operator default)
2. Else runtime.GOARCH (current behavior — preserves #15059's fix)
3. On no child with platform error: retry without platform filter (Option A — graceful fallback)

Happy to send the PR if maintainers prefer.

Contributor guide

Open the contributing guide

Research direction

Start with workflow/controller/entrypoint/container_registry_index.go and inspect the existing controller/entrypoint tests. Reproduce the mismatch with the provided arm64v8/alpine image or an equivalent single-architecture image, then verify that lookup retries without the platform filter when the selected architecture is absent while preserving the preferred-platform behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.