argoproj / argoproj/argo-workflows

Pod failed with exit code 2 even though executed and saved output artifact correctly - no logs from sidecar

Open
#14,577 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

The issue has happened only once, we have no way to reproduce nor any more ideas to find the reason behind this behavior, but maybe you have experienced something similar.

A cron workflow failed, especially one particular pod with exit code 2 even though the execution of logic inside of the pod has been conducted correctly.

As for the design: parallel to this step there were 10 identical steps dealing with similar data, all of them succeeded. The cron has been running daily for couple of months without such incident, both on development and production. This incident happened on production with v3.5.11, identical workflow running on dev cluster at that time passed without issues.

About the logic: it's python code performing operation on data and passing it as output artifact. The artifact has been saved correctly (screenshot below), we can access it and can confirm the data is accurate. Saving the artifact is the last line of code for the script, thus no further logic could cause the error.

![Image](https://github.com/user-attachments/assets/f6a1f06a-07e4-44fb-a5cf-df3be2fc96aa)

Alongside the main container we have sidecar performing kerberos authentication. The only discrepancy between succeeded steps and the failed one we can see, is that we are lacking logs for the sidecar for this pod. Initsidecar and wait containers produced logs correctly, as visible in here:

![Image](https://github.com/user-attachments/assets/ed799740-4ac6-4fdb-b82d-e84ff54db2d6)

Another finding was discovered in the logs from workflow container, where in comparison to succeeded steps, there was kill command sent to the wait container - what did not happen on any of the successful executions.

![Image](https://github.com/user-attachments/assets/6b800e62-c32c-4a8e-94b3-a908d4c75c57)

We would highly appreciate any clues for the reason. Regarding the workaround - would the `retry strategy` prevent this from happening again?

_Apologies for the collage on screenshots. Just wanted to be sure not to expose any sensitive data._

### Version(s)

v3.5.11

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
n/a
```

### Logs from the workflow controller

```text
n/a
```

### Logs from in your workflow's wait container

```text
n/a
```

Contributor guide

Open the contributing guide

Research direction

Start by comparing the failed pod with the ten successful parallel steps, focusing on the sidecar logs, init-sidecar and wait-container behavior, and the reported kill command. The issue provides no reproducible workflow, controller logs, or wait-container logs, so first determine whether additional pod and controller evidence can explain the exit code 2; done would require a reproducible cause or a confirmed fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.