1jehuang / 1jehuang/jcode

[bug] After stopping swarm workers, the daemon keeps replaying their detached commands; processes cannot be killed and the system hangs

Open
#940 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

autonomous: no bug priority: critical regression triage: needs-decision
Dominant language
Rust
Stars
19.9k
Forks
2.3k
Avg merge
2d 7h
Merged PRs (30d)
30

Description

[bug] After stopping swarm workers, the daemon keeps replaying their detached commands; processes cannot be killed and the system hangs

Environment

  • jcode v0.75.3 (fd1ff012c)
  • Linux VM, 2.8 GiB RAM / 6 cores (low-resource environment)
  • display.auto_server_reload = true
  • swarm-deep run_plan (44 nodes), workers running cargo test

Symptom

  1. After swarm stop / swarm cleanup, the commands started by workers (cargo test, the project's test binaries) keep being re-forked: new processes reappear a few seconds after pkill.
  2. The parent chain of the new processes always points at the jcode daemon:
    cargo test --bin <project>   (PID 429698)
    └─ bash -lc eval "$JCODE_RELOAD_DETACH_COMMAND"; ...  (PID 429692)
       └─ jcode-linux-x86_64.bin --provider auto serve  (PID 127135, daemon)
    
  3. When a server reload happens, in-flight worker commands are detached to the daemon (log: Tool 'bg' wait interrupted by server reload ... The underlying operation may still be running). The daemon's recovery mechanism keeps replaying these commands even after the worker session has been stopped.
  4. Result: system load does not drop but rises (11.78 -> 13.66), swap hits 100%, /tmp fills up with test artifacts, and the VM becomes unresponsive. Only manual kill of the whole process chain plus session wipe resolves it.

Repro steps

  1. On a low-resource machine, start a swarm-deep run_plan whose workers run long-running commands (e.g. cargo test).
  2. Trigger a server reload while the commands are in flight (e.g. config hot-update or binary reload).
  3. Call swarm stop / swarm cleanup to stop all workers.
  4. Observe: workers are stopped, but their commands are still running under the daemon and new instances keep being forked.

Expected behavior

  • Stopping a worker should cascade-cancel all of its detached commands; they must not be replayed.
  • The reload-recovery mechanism should check whether the owning session is still alive before replaying a detached command.

Additional observations

  • ~/.jcode/reload-recovery/ only contains old records from previous days, so this replay path appears to be a different recovery path (JCODE_RELOAD_DETACH_COMMAND), not the same one that writes to reload-recovery/.
  • The impact is amplified on low-resource machines: with high worker concurrency (20 in this case), many compile/test commands are replayed at once and take the whole system down.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reload scenario with a swarm-deep run_plan whose workers run cargo test, then trace the daemon path triggered by JCODE_RELOAD_DETACH_COMMAND. Compare behavior before and after swarm stop or swarm cleanup. Done means detached worker commands are cancelled and no longer replayed once their owning worker session has stopped.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.