argoproj / argoproj/argo-workflows
argo logs --follow with --selector hangs until workflow completes
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
When running the following command against a multi-step workflow:
```bash
argo logs my-wf --selector=my-label --follow
```
I receive the expected logged output, but the command does not resolve until the entire workflow is complete.
My use-case in which this becomes problematic is: I have a multi-step workflow in which several of the steps are run in parallel. I'd like to be able to retrieve logs from individual steps while the workflow is running so that the logged output from the parallel steps is not interleaved.
While I am able to get the logs from individual Pods, after calling `argo logs --selector=... --follow ...` once, this command hangs until the entire workflow is complete and my subsequent `argo logs ...` calls must wait until the workflow has completed.
In my deployment pipeline, this ends up giving the impression that the workflow has stalled after the first step, when, in reality, the workflow is running as expected.
```text
argo logs test-123 --selector=my-label=serial --follow
test-123-serial-tpl-3222931304: this is output from the serial container
test-123-serial-tpl-3222931304: time="2025-04-08T19:14:54.246Z" level=info msg="sub-process exited" argo=true error=""
# 60 second wait ...
argo logs test-123 --selector=my-label=parallel-1 --follow
test-123-parallel-tpl-1-882714198: this is output from parallel container one
test-123-parallel-tpl-1-882714198: this is output from parallel container one
test-123-parallel-tpl-1-882714198: this is output from parallel container one
test-123-parallel-tpl-1-882714198: this is output from parallel container one
...
# No wait ...
argo logs test-123 --selector=my-label=parallel-2 --follow
test-123-parallel-tpl-2-865936579: this is output from parallel container two
test-123-parallel-tpl-2-865936579: this is output from parallel container two
test-123-parallel-tpl-2-865936579: this is output from parallel container two
test-123-parallel-tpl-2-865936579: this is output from parallel container two
test-123-parallel-tpl-2-865936579: this is output from parallel container two
...
```
Here, I would expect that, once the selected Pod has completed, the command would resolve immediately.
### Version(s)
v3.6.2, latest
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: test-123
spec:
entrypoint: test
serviceAccountName: default
templates:
- name: test
steps:
- - name: serial
template: serial-tpl
- - name: parallel-1
template: parallel-tpl-1
- name: parallel-2
template: parallel-tpl-2
- name: serial-tpl
metadata:
labels:
my-label: serial
container:
image: bash:5.2
command: [bash, '-c']
args:
- |
echo "this is output from the serial container"
- name: parallel-tpl-1
metadata:
labels:
my-label: parallel-1
container:
image: bash:5.2
command: [bash, '-c']
args:
- |
for i in {1..60}; do echo "this is output from parallel container one"; sleep 1s; done
- name: parallel-tpl-2
metadata:
labels:
my-label: parallel-2
container:
image: bash:5.2
command: [bash, '-c']
args:
- |
for i in {1..60}; do echo "this is output from parallel container two"; sleep 1s; done
```
### Logs from the workflow controller
```text
Omitted, no errors here.
```
### Logs from in your workflow's wait container
```text
Omitted, no errors here.
```
Contributor guide
Research direction
Start by reproducing the issue with the provided multi-step workflow and trace the `argo logs --selector --follow` CLI path. Check how selected Pod completion is handled while the workflow remains active; done means the command returns when the selected Pod completes, without waiting for the full workflow, with regression coverage for parallel steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100