argoproj / argoproj/argo-workflows
Executor tries to use imagePullSecrets to pull a container image even if anonymous pull is enabled
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))
### What happened/what you expected to happen?
Kubernetes pulls the same images without using imagePullSecrets if anonymous pull is allowed.
Argo executor shall also pull a container image (to check the cmd/args value) without using imagePullSecrets if anonymous pull is enabled.
### Version
3.4.1
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
The issue is related to a private images
```
### Logs from the workflow controller
```sh
kubectl logs -n argo deploy/argo-helm-argo-workflows-workflow-controller | grep acm-adhoc-bps-db-version-1665584107
```
```
time="2022-10-12T16:17:29.499Z" level=info msg="Processing workflow" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.589Z" level=info msg="Updated phase -> Running" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.589Z" level=info msg="DAG node acm-adhoc-bps-db-version-1665584107 initialized Running" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.590Z" level=info msg="All of node acm-adhoc-bps-db-version-1665584107.db-version dependencies [] completed" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.591Z" level=info msg="DAG node acm-adhoc-bps-db-version-1665584107-3268391737 initialized Running" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.592Z" level=info msg="All of node acm-adhoc-bps-db-version-1665584107.db-version.db-version-task dependencies [] completed" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.594Z" level=info msg="Pod node acm-adhoc-bps-db-version-1665584107-1331475456 initialized Pending" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.605Z" level=error msg="Mark error node" error="failed to look-up entrypoint/cmd for image \"artifacts-scm.dstcorp.net/algo-docker/acm/bps:acm-5.5.3-00\", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: secrets \"acm-registry-creds\" not found" namespace=acmtmp nodeName=acm-adhoc-bps-db-version-1665584107.db-version.db-version-task workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.605Z" level=info msg="node acm-adhoc-bps-db-version-1665584107-1331475456 phase Pending -> Error" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.605Z" level=info msg="node acm-adhoc-bps-db-version-1665584107-1331475456 message: failed to look-up entrypoint/cmd for image \"artifacts-scm.dstcorp.net/algo-docker/acm/bps:acm-5.5.3-00\", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: secrets \"acm-registry-creds\" not found" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.605Z" level=info msg="node acm-adhoc-bps-db-version-1665584107-1331475456 finished: 2022-10-12 16:17:29.605737508 +0000 UTC" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.605Z" level=error msg="Mark error node" error="task 'acm-adhoc-bps-db-version-1665584107.db-version.db-version-task' errored: failed to look-up entrypoint/cmd for image \"artifacts-scm.dstcorp.net/algo-docker/acm/bps:acm-5.5.3-00\", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: secrets \"acm-registry-creds\" not found" namespace=acmtmp nodeName=acm-adhoc-bps-db-version-1665584107.db-version.db-version-task workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.605Z" level=info msg="node acm-adhoc-bps-db-version-1665584107-1331475456 message: task 'acm-adhoc-bps-db-version-1665584107.db-version.db-version-task' errored: failed to look-up entrypoint/cmd for image \"artifacts-scm.dstcorp.net/algo-docker/acm/bps:acm-5.5.3-00\", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: secrets \"acm-registry-creds\" not found" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.608Z" level=info msg="Outbound nodes of acm-adhoc-bps-db-version-1665584107-3268391737 set to [acm-adhoc-bps-db-version-1665584107-1331475456]" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.608Z" level=info msg="node acm-adhoc-bps-db-version-1665584107-3268391737 phase Running -> Error" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.608Z" level=info msg="node acm-adhoc-bps-db-version-1665584107-3268391737 finished: 2022-10-12 16:17:29.608557778 +0000 UTC" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.608Z" level=info msg="Checking daemoned children of acm-adhoc-bps-db-version-1665584107-3268391737" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="Outbound nodes of acm-adhoc-bps-db-version-1665584107 set to [acm-adhoc-bps-db-version-1665584107-1331475456]" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="node acm-adhoc-bps-db-version-1665584107 phase Running -> Error" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="node acm-adhoc-bps-db-version-1665584107 finished: 2022-10-12 16:17:29.61131521 +0000 UTC" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="Checking daemoned children of acm-adhoc-bps-db-version-1665584107" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="TaskSet Reconciliation" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg=reconcileAgentPod namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="Updated phase Running -> Error" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="Marking workflow completed" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="Marking workflow as pending archiving" namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.611Z" level=info msg="Checking daemoned children of " namespace=acmtmp workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.617Z" level=info msg="cleaning up pod" action=deletePod key=acmtmp/acm-adhoc-bps-db-version-1665584107-1340600742-agent/deletePod
time="2022-10-12T16:17:29.624Z" level=info msg="Workflow update successful" namespace=acmtmp phase=Error resourceVersion=104803467 workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.628Z" level=info msg="archiving workflow" namespace=acmtmp uid=6f8c2dca-17f4-4081-b564-b3f82720a28e workflow=acm-adhoc-bps-db-version-1665584107
time="2022-10-12T16:17:29.659Z" level=info msg="Queueing Error workflow acmtmp/acm-adhoc-bps-db-version-1665584107 for delete in 5m0s due to TTL"
```
### Logs from in your workflow's wait container
No wait logs available, seems argo did not come that far.
Contributor guide
Research direction
Start by tracing the executor path responsible for looking up an image's entrypoint/cmd, using the error mentioning the emissary executor and imagePullSecrets. Compare that image lookup with Kubernetes' anonymous-pull behavior; done means an image configured for anonymous pulling is checked without requiring the unavailable secret, with coverage for the reported failure mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100