pipe-cd / pipe-cd/pipecd

Zombie processes and goroutine leak in ScriptRun / CustomSync executors on cancellation

Open
#6,734 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/piped kind/bug
Dominant language
Go
Stars
1.4k
Forks
364
Avg merge
1d 4h
Merged PRs (30d)
84

Description

What happened:
When a deployment is cancelled or times out during a ScriptRun, CustomSync, or Kubernetes rollback stage, the Piped agent stops tracking the stage but fails to terminate the underlying shell process. This results in orphaned "zombie" processes (e.g., kubectl, terraform, sleep) and leaked goroutines on the Piped agent host. These orphaned processes continue to run and mutate cluster state even after a rollback has been initiated, leading to potential data corruption and agent resource exhaustion.

What you expected to happen:
When a stage is cancelled or times out, the Piped agent should immediately and definitively terminate the underlying OS process tree associated with that stage's command. This ensures that no unauthorized mutations occur after cancellation and that agent resources are correctly reclaimed.

How to reproduce it:

  1. Create a PipeCD application with a ScriptRun stage that executes a long-running command (e.g., run: sleep 300).
  2. Trigger a deployment for this application.
  3. Once the ScriptRun stage is active, click the "Cancel" button in the PipeCD web UI.
  4. Observe the PipeCD UI and logs; the stage will be reported as CANCELLED.
  5. Check the process list on the Piped agent host (e.g., ps aux | grep sleep). You will find that the sleep process is still running in the background.

Environment:

  • piped version: master / latest
  • control-plane version: master / latest
  • Others: This is a platform-agnostic bug affecting any Piped agent utilizing ScriptRun or CustomSync executors.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace cancellation and timeout handling for the ScriptRun and CustomSync executors, including Kubernetes rollback, and follow how the underlying shell process is started and tracked. Reproduce the issue with a long-running sleep 300 command and inspect the agent process list and logs after cancellation. Done means the process tree terminates and no goroutines or child processes remain after the stage is cancelled or times out.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.