argoproj / argoproj/argo-workflows

Wait Container stucks which cause workflow keep running

Open
#11,892 10 comments 1 reaction 0 assignees View on GitHub
area/executor P3
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

We found sometimes workflow keep running for a long time. We investigated and found the root cause was the wait container of single pod in workflow stuck as the following:
![image](https://github.com/argoproj/argo-workflows/assets/64390829/07077807-b764-4aa6-acb9-02713510ed69)
We are very struggle about it. We don't know which caused wait container stuck. It's very hard to recure this bug.

### Version

3.4.4

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
No
```

### Logs from the workflow controller

```text
No
```

### Logs from in your workflow's wait container

```text
No
```

Contributor guide

Open the contributing guide

Research direction

No source file, test, reproducible workflow, controller log, or wait-container log is provided. Start by reproducing the stuck wait container on Argo Workflows 3.4.4 and collect the workflow pod state plus both controller and wait-container logs; done means identifying a repeatable cause and a verified fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.