argoproj / argoproj/argo-workflows
Using hostPID: true + nsenter in argoexec causes host processes (e.g. sshd, systemd-resolved) to be killed when a Pod is deleted
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
Our workflow executor (the “emissary”/argoexec container) runs with hostPID: true. Inside the container, we launch host-level commands by calling something like:
`nsenter --target 1 --mount --uts --ipc --net --pid -- /usr/bin/some-binary …
The intention is to execute operations directly on the node’s PID namespace. However, as soon as Kubernetes deletes the Pod, the node’s critical processes (e.g. sshd, systemd-resolved) also receive SIGTERM/SIGKILL and get shut down.
### Version(s)
v3.6.5
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
apiVersion: v1
kind: Pod
metadata:
name: argoexec-ssh-mp-test02
spec:
containers:
- image: 10.10.16.36:31373/ubuntu-argoexec:latest
imagePullPolicy: IfNotPresent
name: main
command:
- /usr/bin/argoexec
- emissary
- --loglevel
- warning
- --log-format
- text
- --gloglevel
- "0"
- --
- nsenter
- -t
- "1"
- -m
- -u
- -i
- -n
- -p
- --
- bash
- -c
args:
- sleep infinity
env:
- name: ARGO_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ARGO_POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: ARGO_TEMPLATE
value: '{}'
resources: {}
securityContext:
privileged: true
volumeMounts:
- name: host
mountPath: /host
dnsPolicy: ClusterFirstWithHostNet
enableServiceLinks: true
hostPID: true
nodeName: mp-test02
restartPolicy: Never
schedulerName: default-scheduler
volumes:
- name: host
hostPath:
path: /
```
### Logs from the workflow controller
```text
-
```
### Logs from in your workflow's wait container
```text
-
```
Contributor guide
Research direction
Start at the argoexec emissary entry point and reproduce the supplied Pod with hostPID and nsenter. Investigate what happens to host processes when Kubernetes deletes the Pod, then verify that Pod deletion no longer sends termination signals to unrelated host processes using the same reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, linux
- Domain
- infrastructure, operating-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100