Zombie processes and goroutine leak in ScriptRun / CustomSync executors on cancellation
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 364
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 84
Description
What happened:
When a deployment is cancelled or times out during a ScriptRun, CustomSync, or Kubernetes rollback stage, the Piped agent stops tracking the stage but fails to terminate the underlying shell process. This results in orphaned "zombie" processes (e.g., kubectl, terraform, sleep) and leaked goroutines on the Piped agent host. These orphaned processes continue to run and mutate cluster state even after a rollback has been initiated, leading to potential data corruption and agent resource exhaustion.
What you expected to happen:
When a stage is cancelled or times out, the Piped agent should immediately and definitively terminate the underlying OS process tree associated with that stage's command. This ensures that no unauthorized mutations occur after cancellation and that agent resources are correctly reclaimed.
How to reproduce it:
- Create a PipeCD application with a ScriptRun stage that executes a long-running command (e.g.,
run: sleep 300). - Trigger a deployment for this application.
- Once the ScriptRun stage is active, click the "Cancel" button in the PipeCD web UI.
- Observe the PipeCD UI and logs; the stage will be reported as
CANCELLED. - Check the process list on the Piped agent host (e.g.,
ps aux | grep sleep). You will find that the sleep process is still running in the background.
Environment:
- piped version: master / latest
- control-plane version: master / latest
- Others: This is a platform-agnostic bug affecting any Piped agent utilizing ScriptRun or CustomSync executors.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace cancellation and timeout handling for the ScriptRun and CustomSync executors, including Kubernetes rollback, and follow how the underlying shell process is started and tracked. Reproduce the issue with a long-running sleep 300 command and inspect the agent process list and logs after cancellation. Done means the process tree terminates and no goroutines or child processes remain after the stage is cancelled or times out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100