HTTP Template: exit handler on a retry ignores or gets stuck

Open

#11,072 opened on May 11, 2023 2 comments 6 reactions 0 assignees View on GitHub

area/exit-handler area/hooks area/retry-manual area/templates/http P1

Dominant language: Go
Stars: 17k
Forks: 3.7k
Avg merge: 1d 20h
Merged PRs (30d): 138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

We've switched from `container` templates to `http` template and got unexpected behaviour on retry (both in a pod and a archived).

There are 2 API for the retry:
-`argo retry workflow-name`
-`argo archive retry workflow-name`
We use both: first when the pod still in k8s, and second when the pod GC (archived).

Actual behaviour:
- On `argo retry workflow-name` on exit handler is ignored
- On `argo archive retry workflow-name` on exit handler is stuck infinity in running state

Expected behaviour:
- On `argo retry workflow-name` on exit handle should run anyway as it's in a container template
- On `argo archive retry workflow-name` on exit handler should not stuck and run anyway as it's in a container template

### Version

3.4.7 (latest)

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: http-template-
spec:
entrypoint: main
onExit: exit-handler
templates:
- name: main
http:
method: POST
url: https://www.google.com/foo/bar
- name: exit-handler
http:
method: GET
url: https://reqres.in/api/users/2
```

### Logs from the workflow controller

```text
POD RETRY LOGS:

ARCHVIED RETRY:
{"level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2023-05-11T13:38:09.223Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2023-05-11T13:38:09.224Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"HTTP node http-template-xqbwx initialized Pending","namespace":"customer-workflows","time":"2023-05-11T13:38:09.224Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:09.224Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Creating TaskSet","namespace":"customer-workflows","time":"2023-05-11T13:38:09.224Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:09.228Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Created Agent pod","namespace":"customer-workflows","podName":"http-template-xqbwx-1340600742-agent","time":"2023-05-11T13:38:09.248Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"updateAgentPodStatus","namespace":"customer-workflows","time":"2023-05-11T13:38:09.248Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"assessAgentPodStatus","namespace":"customer-workflows","podName":"http-template-xqbwx-1340600742-agent","time":"2023-05-11T13:38:09.248Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"59523","time":"2023-05-11T13:38:09.257Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2023-05-11T13:38:19.224Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2023-05-11T13:38:19.225Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"updateAgentPodStatus","namespace":"customer-workflows","time":"2023-05-11T13:38:19.225Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"assessAgentPodStatus","namespace":"customer-workflows","podName":"http-template-xqbwx-1340600742-agent","time":"2023-05-11T13:38:19.225Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:19.225Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:19.225Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2023-05-11T13:38:32.052Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2023-05-11T13:38:32.053Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"updateAgentPodStatus","namespace":"customer-workflows","time":"2023-05-11T13:38:32.053Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"assessAgentPodStatus","namespace":"customer-workflows","podName":"http-template-xqbwx-1340600742-agent","time":"2023-05-11T13:38:32.053Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:32.053Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:32.053Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"59568","time":"2023-05-11T13:38:32.056Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2023-05-11T13:38:42.062Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2023-05-11T13:38:42.062Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"updateAgentPodStatus","namespace":"customer-workflows","time":"2023-05-11T13:38:42.062Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"assessAgentPodStatus","namespace":"customer-workflows","podName":"http-template-xqbwx-1340600742-agent","time":"2023-05-11T13:38:42.062Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:42.063Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:42.063Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2023-05-11T13:38:42.063Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:42.063Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:42.063Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"59586","time":"2023-05-11T13:38:42.067Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2023-05-11T13:38:52.068Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"updateAgentPodStatus","namespace":"customer-workflows","time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"assessAgentPodStatus","namespace":"customer-workflows","podName":"http-template-xqbwx-1340600742-agent","time":"2023-05-11T13:38:52.069Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2023-05-11T13:38:52.069Z","workflow":"http-template-xqbwx"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"59601","time":"2023-05-11T13:38:52.073Z","workflow":"http-template-xqbwx"}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=http-template-xqbwx,workflow.argoproj.io/phase!=Succeeded

No resources found in argo namespace.
```

Contributor guide

Open the contributing guide

Research direction

Start with the provided HTTP-template workflow and reproduce both `argo retry workflow-name` and `argo archive retry workflow-name` paths, using the workflow-controller logs as the entry point. Trace how the onExit `exit-handler` is scheduled after retry and archiving; done means it runs once and does not remain stuck in Running for either path.

Written by the indexing model from the issue text.

Assessment

Tech stack: go, kubernetes
Domain: backend, distributed-systems
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Mostly clear
Newbie friendliness: 35/100

HTTP Template: exit handler on a retry ignores or gets stuck

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox