argoproj / argoproj/argo-workflows

Using hooks, stop workflow, keeps running.

Open
#9,640 6 comments 0 reactions 0 assignees View on GitHub
area/hooks P3
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

When I am using hooks, when I stop this workflow, it does not stop but keeps running.

Workflow configuration file:

https://github.com/argoproj/argo-workflows/blob/45730a9cdeb588d0e52b1ac87b6e0ca391a95a81/examples/life-cycle-hooks-tmpl-level.yaml

I have stopped, but the state of the hooks has been 'PHASE Pending'

![Snipaste_2022-09-21_11-03-20.png](https://s2.loli.net/2022/09/21/VLiPTnvbDCQG3Fu.png)

### Version

v3.3.5

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: lifecycle-hook-tmpl-level-
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: step-1
hooks:
exit:
# Expr will not support `-` on variable name. Variable should wrap with `[]`
expression: steps["step-1"].status == "Running"
template: http
success:
expression: steps["step-1"].status == "Succeeded"
template: http
template: echo
- - name: step2
hooks:
exit:
expression: steps.step2.status == "Running"
template: http
success:
expression: steps.step2.status == "Succeeded"
template: http
template: echo

- name: echo
container:
image: alpine:3.6
command: [sh, -c]
args: ["sleep 30 && echo \"it was heads\""]

- name: http
http:
# url: http://dummy.restapiexample.com/api/v1/employees
url: "https://raw.githubusercontent.com/argoproj/argo-workflows/4e450e250168e6b4d51a126b784e90b11a0162bc/pkg/apis/workflow/v1alpha1/generated.swagger.json"
```

### Logs from the workflow controller

```
[root@k8s-master01 ~]# kubectl logs -n argo deploy/workflow-controller | grep lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.089Z" level=info msg="Processing workflow" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.089Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.089Z" level=info msg="task-result changed" namespace=argo nodeID=lifecycle-hook-tmpl-level-frx69-2640744148 workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.092Z" level=info msg="Running OnExit handler" lifeCycleHook="&LifecycleHook{Template:http,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Expression:steps[\"step-1\"].status == \"Running\",}" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.093Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-frx69-2601770046 not yet completed" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.093Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.093Z" level=info msg=reconcileAgentPod namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:27.120Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=7912549 workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:37.030Z" level=info msg="Processing workflow" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:37.030Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:37.031Z" level=info msg="Running OnExit handler" lifeCycleHook="&LifecycleHook{Template:http,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Expression:steps[\"step-1\"].status == \"Running\",}" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:37.031Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-frx69-2601770046 not yet completed" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:37.031Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:01:37.031Z" level=info msg=reconcileAgentPod namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:21:36.997Z" level=info msg="Processing workflow" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:21:36.997Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:21:36.998Z" level=info msg="Running OnExit handler" lifeCycleHook="&LifecycleHook{Template:http,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Expression:steps[\"step-1\"].status == \"Running\",}" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:21:36.998Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-frx69-2601770046 not yet completed" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:21:36.998Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
time="2022-09-21T03:21:36.998Z" level=info msg=reconcileAgentPod namespace=argo workflow=lifecycle-hook-tmpl-level-frx69
```

### Logs from in your workflow's wait container

```
[root@k8s-master01 ~]# kubectl logs -c wait -l workflows.argoproj.io/workflow=lifecycle-hook-tmpl-level-frx69,workflow.argoproj.io/phase!=Succeeded
No resources found in default namespace.
```

Contributor guide

Open the contributing guide

Research direction

Start with examples/life-cycle-hooks-tmpl-level.yaml and trace the workflow-controller handling shown in the logs when a workflow is stopped. Reproduce the pending exit hook state with the supplied workflow and inspect how the wait container and hook completion are reconciled. Done means stopping the workflow no longer leaves it running because the hook remains pending.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.