argoproj / argoproj/argo-workflows
`AgentPod` misses serviceaccounts of plugin in controller namespace
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I can confirm the issue exists when I tested with `:latest`
- [X] I have searched existing issues and could not find a match for this bug
- [X] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened/what did you expect to happen?
Related discussion: https://github.com/argoproj/argo-workflows/discussions/12566
Controller runs in namespace: `argo`, there is a plugin named `khaos-executor-plugin` and a serviceaccount named `khaos-executor-plugin`.
The sample workflow runs in namespace: `khaos-workflow`, there is a plugin named `hello-executor-plugin` and a serviceaccount named `hello-executor-plugin`.
```shell
argo get hello-7t78h -n khaos-workflow
Name: hello-7t78h
Namespace: khaos-workflow
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Error
Message: serviceaccounts "khaos-executor-plugin" not found
Conditions:
Completed True
PodRunning False
Created: Wed Feb 28 17:32:52 +0800 (2 hours ago)
Started: Wed Feb 28 17:39:01 +0800 (2 hours ago)
Finished: Wed Feb 28 17:39:01 +0800 (2 hours ago)
Duration: 0 seconds
Progress: 0/1
STEP TEMPLATE PODNAME DURATION MESSAGE
◷ hello-7t78h main
```
### Version
v3.4.14
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-
spec:
entrypoint: main
templates:
- name: main
plugin:
hello: { }
```
### Logs from the workflow controller
```text
time="2024-02-28T09:39:01.368Z" level=info msg="Task-result reconciliation" namespace=khaos-workflow numObjs=0 workflow=hello-7t78h
time="2024-02-28T09:39:01.368Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.368Z" level=warning msg="[DEBUG] boundaryID was nil" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.368Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.368Z" level=info msg="Plugin node hello-7t78h initialized Pending" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.368Z" level=info msg="TaskSet Reconciliation" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.368Z" level=info msg="Creating TaskSet" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.438Z" level=info msg=reconcileAgentPod namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.458Z" level=error msg="error in agent pod reconciliation" error="serviceaccounts \"khaos-executor-plugin\" not found" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.458Z" level=info msg="Updated phase Running -> Error" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.458Z" level=info msg="Updated message -> serviceaccounts \"khaos-executor-plugin\" not found" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.458Z" level=info msg="Marking workflow completed" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.458Z" level=info msg="Marking workflow as pending archiving" namespace=khaos-workflow workflow=hello-7t78h
time="2024-02-28T09:39:01.458Z" level=info msg="Workflow to be dehydrated" Workflow Size=1516
time="2024-02-28T09:39:01.466Z" level=info msg="cleaning up pod" action=deletePod key=khaos-workflow/hello-7t78h-1340600742-agent/deletePod
time="2024-02-28T09:39:01.468Z" level=info msg="Workflow update successful" namespace=khaos-workflow phase=Error resourceVersion=85399559051 workflow=hello-7t78h
time="2024-02-28T09:39:01.491Z" level=info msg="archiving workflow" namespace=khaos-workflow uid=5c4aa38d-4d94-4def-b189-b6908d3133e4 workflow=hello-7t78h
time="2024-02-28T09:39:01.513Z" level=info msg="Queueing Error workflow khaos-workflow/hello-7t78h for delete in 72h0m0s due to TTL"
time="2024-02-28T09:39:56.542Z" level=info msg="Alloc=17143 TotalAlloc=4920922 Sys=61565 NumGC=947 Goroutines=282"
time="2024-02-28T09:40:17.314Z" level=info msg="Performing periodic workflow GC"
time="2024-02-28T09:40:17.316Z" level=info msg="Deleting old offloads that are not live" len_wfs=0
time="2024-02-28T09:40:17.316Z" level=info msg="Workflow GC finished"
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Start by tracing the controller's reconcileAgentPod path using the provided workflow and controller logs, focusing on how the plugin serviceaccount is selected across the argo and khaos-workflow namespaces. Reproduce the AgentPod failure with the sample workflow, then verify that the expected plugin serviceaccount is found and the workflow no longer errors with a missing serviceaccount.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100