[bug] After stopping swarm workers, the daemon keeps replaying their detached commands; processes cannot be killed and the system hangs
Open
Nobody has claimed this yet.
autonomous: no
bug
priority: critical
regression
triage: needs-decision
- Dominant language
- Rust
- Stars
- 19.9k
- Forks
- 2.3k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 30
Description
[bug] After stopping swarm workers, the daemon keeps replaying their detached commands; processes cannot be killed and the system hangs
Environment
- jcode v0.75.3 (fd1ff012c)
- Linux VM, 2.8 GiB RAM / 6 cores (low-resource environment)
display.auto_server_reload = true- swarm-deep
run_plan(44 nodes), workers runningcargo test
Symptom
- After
swarm stop/swarm cleanup, the commands started by workers (cargo test, the project's test binaries) keep being re-forked: new processes reappear a few seconds afterpkill. - The parent chain of the new processes always points at the jcode daemon:
cargo test --bin <project> (PID 429698) └─ bash -lc eval "$JCODE_RELOAD_DETACH_COMMAND"; ... (PID 429692) └─ jcode-linux-x86_64.bin --provider auto serve (PID 127135, daemon) - When a server reload happens, in-flight worker commands are detached to the daemon (log:
Tool 'bg' wait interrupted by server reload ... The underlying operation may still be running). The daemon's recovery mechanism keeps replaying these commands even after the worker session has been stopped. - Result: system load does not drop but rises (11.78 -> 13.66), swap hits 100%, /tmp fills up with test artifacts, and the VM becomes unresponsive. Only manual kill of the whole process chain plus session wipe resolves it.
Repro steps
- On a low-resource machine, start a swarm-deep
run_planwhose workers run long-running commands (e.g.cargo test). - Trigger a server reload while the commands are in flight (e.g. config hot-update or binary reload).
- Call
swarm stop/swarm cleanupto stop all workers. - Observe: workers are stopped, but their commands are still running under the daemon and new instances keep being forked.
Expected behavior
- Stopping a worker should cascade-cancel all of its detached commands; they must not be replayed.
- The reload-recovery mechanism should check whether the owning session is still alive before replaying a detached command.
Additional observations
~/.jcode/reload-recovery/only contains old records from previous days, so this replay path appears to be a different recovery path (JCODE_RELOAD_DETACH_COMMAND), not the same one that writes toreload-recovery/.- The impact is amplified on low-resource machines: with high worker concurrency (20 in this case), many compile/test commands are replayed at once and take the whole system down.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reload scenario with a swarm-deep run_plan whose workers run cargo test, then trace the daemon path triggered by JCODE_RELOAD_DETACH_COMMAND. Compare behavior before and after swarm stop or swarm cleanup. Done means detached worker commands are cancelled and no longer replayed once their owning worker session has stopped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100