Orphan process cleanup misses processes created during its own snapshot-then-kill pass

Describe the bug

Cleaning up orphan processes takes a single process-table snapshot and then inspects and kills from that snapshot. A process created after the snapshot is never inspected, so it is never terminated, even though it carries the job's RUNNER_TRACKING_ID and is exactly what the cleanup exists to collect. If its parent is in the snapshot, the parent is killed moments later in the same pass and the new process is left reparented to init, outliving the job indefinitely.

The window is not instantaneous, because the pass does a per-process /proc/<pid>/environ read between the snapshot and the kills. In src/Runner.Worker/JobExtension.cs#L868-L915, pinned at b7fd7da:

context.Output("Cleaning up orphan processes");

// Only check environment variable for any process that doesn't run before we invoke our process.
Dictionary<int, Process> currentProcesses = SnapshotProcesses();   // one snapshot
foreach (var proc in currentProcesses)
{
    ...
    lookupId = proc.Value.GetEnvironmentVariable(HostContext, Constants.ProcessTrackingId);
    ...
    proc.Value.Kill();                                             // kill, same pass
}

SnapshotProcesses() is a plain Process.GetProcesses(), and on Linux GetEnvironmentVariable reads and parses the whole of /proc/<pid>/environ for each candidate. Those reads are serial and inside the same loop as the kills, so the interval between "which processes exist" and "this one is dead" is milliseconds to hundreds of milliseconds wide, scaling with how many processes the job started. There is no second pass after the kills.

To Reproduce

What I actually observed were two occurrences on cancelled jobs, described under "Job Log Output" below. The following is the smallest shape I believe reproduces them, offered as a sketch: I have not run this exact workflow. Each iteration creates a new process, so with a pass a few hundred milliseconds wide, at least one child should land after the snapshot.

1. Run this job on a self-hosted Linux runner:

jobs:
  leak:
    runs-on: [self-hosted, linux]
    steps:
      - run: |
          while true; do
            sh -c 'exec sleep 3600' &
            sleep 0.05
          done

2. Cancel the job while the step is running.
3. On the host, look for survivors: sleep 3600 processes with PPID 1, still running after the job has ended. In the job log, the terminate list stops below their pids.

Expected behavior

The cleanup collects every process carrying this job's RUNNER_TRACKING_ID, including ones created while the cleanup itself is running. Today it collects only those that existed at the moment of its snapshot.

Runner Version and Platform

Runner 2.336.0, self-hosted, Linux x64. Two hosts, one bare metal and one WSL2, both affected. process.clean at its default (enabled). Both occurrences were on cancelled jobs; the same jobs completing normally leave nothing behind.

What's not working?

The leaked process keeps running indefinitely, reparented to init, holding a bound loopback socket. Nothing on the host reclaims it: not the next job, and not the cleanup that missed it, since it will not be in that job's tracking set either.

The workload is generic: a Node CLI that spawns a long-lived native server as a child and supervises it over a control pipe. The CLI is killed by the pass; the server is not, and with its supervisor gone nothing remains that could ask it to stop. Any "CLI spawns a daemon-shaped child" tool has the same exposure; a build tool's background daemon would behave the same way.

Job Log Output

Times below are relative to the Cleaning up orphan processes line, and identifying details from our environment are omitted, so this is a summary of those log lines rather than a paste of them.

	pass duration	terminated	leaked process created	its pid vs the highest pid the pass named
run A	245 ms	9 processes	T+196 ms, i.e. inside the pass	+996
run B	141 ms	9 processes	within the same second as the pass	+402

In both cases the leaked process's own parent appears in the terminate list. The pid ordering is the part I would point at: the leaked pid is above every pid the pass named, so it did not exist when the snapshot was taken. Both processes were still running hours later.

Runner and Worker's Diagnostic Logs

Not included. The _diag logs for these two runs have since aged out of our retention, and the job log lines summarized above plus the code path identify the sequence without them.

Possible fixes

1. Repeat the pass until it terminates nothing (bounded, say 3 iterations). Closes the window rather than narrowing it, and is contained entirely within the existing block.
2. Kill(entireProcessTree: true) instead of Kill(), so a tagged process's descendants are collected at kill time even when they postdate the snapshot. Smaller, but partial: a descendant whose parent was already killed earlier in the same pass is still missed, so this pairs with (1) rather than replacing it.
3. Scope the job to a cgroup on Linux and kill the cgroup. Removes both the enumeration and the /proc/<pid>/environ cost entirely. Largest change.

Why the usual workarounds don't cover it

A cleanup trap in the step is the natural answer, and it is the thing that cannot run here: on cancellation the step is killed without a chance to trap, which is precisely when this fires. Nor can the leaked process shut itself down, since the parent it was supervised by is gone. Disabling process.clean is the opposite of what is wanted.

For our part this is already handled: a periodic sweep on our hosts reaps processes whose supervisor is gone, so nothing is blocked on this. Filing it because the gap is in the runner's cleanup rather than in the tools it supervises, and the fix looks small.

Start in src/Runner.Worker/JobExtension.cs around lines 868-915 and trace SnapshotProcesses(), the per-process environment lookup, and the kill loop. Reproduce or inspect the cancellation cleanup path on Linux, then verify that processes carrying the job's RUNNER_TRACKING_ID and created during cleanup are also terminated, with no survivors after the pass.