anthropics / anthropics/claude-plugins-official

telegram: orphaned server.ts processes accumulate and burn CPU (27 orphans, 131% CPU); all four shutdown paths share one event loop

Offen
#5,745 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
36.2k
Forks
4.1k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Summary

On a host that runs several concurrent Claude Code sessions, `telegram` plugin `server.ts` processes accumulate indefinitely. I found **31 live `bun server.ts` processes, 27 of them orphaned**, the oldest **6.8 days** old. Together they were consuming **131.7% CPU on a 2-core box** — about two thirds of the machine — with a sustained load average of 40-50.

The practical effect is not just the CPU: `fork()` cost went from ~1.3 ms to ~11 ms, so every unrelated shell-heavy job on the host slowed down ~10x. That is how I found it — a cron job that normally takes under 3 minutes was timing out at 400 s.

Version: plugin `telegram` 0.0.7 (`claude-channel-telegram`), Bun, Linux.

## Why the existing shutdown paths don't fire

`server.ts` has four independent shutdown mechanisms, and that looks robust:

1. PID file (`bot.pid`) — a newly starting server sends `SIGTERM` to the previous holder
2. `process.stdin.on('end'|'close')` → `shutdown()` — the MCP transport pipe dies with the CLI
3. an orphan watchdog, `setInterval(..., 5000)`, checking `stdin.destroyed || stdin.readableEnded`
4. `process.on('SIGTERM'|'SIGINT'|'SIGHUP')` → `shutdown()`, with a 2 s force-exit

**All four run on the same event loop.** If the loop is saturated, none of them execute — they are effectively one mechanism, not four. This is not speculative: the 27 orphans **ignored `SIGTERM` entirely** and only died on `SIGKILL`.

Mechanism 1 has a second limitation: it only targets the single PID recorded in `bot.pid`, so it can never clean up N accumulated orphans — only the most recent one.

## Evidence that the orphans are spinning, not idle

Measured as a delta of `utime+stime` from `/proc//stat` (not `ps %CPU`, which is an average since process start and hides this):

| | CPU |
|---|---|
| healthy servers with a live parent | **0.0-0.3%** |
| the 27 orphans | **~4.2% each**, sustained for days |

A healthy server is essentially free. The orphans were burning CPU continuously.

## Likely cause of the spin — not verified

The most probable explanation is a retry loop against **409 Conflict** on `getUpdates`: Telegram allows exactly one `getUpdates` consumer per token, so with N processes sharing a token, N-1 get a permanent 409. The comment above the PID-file logic in `server.ts` describes exactly this scenario as the thing the PID file exists to prevent.

I could not confirm it: the server's stderr goes to the CLI and is not persisted anywhere on disk, and calling `getUpdates` myself to check would have stolen the slot from the one live bot. **Capturing the stderr of an orphan before killing it is the missing piece.** If that is the cause, the failure is self-reinforcing: more orphans → more 409s → more CPU burned → less chance any of them ever processes a signal → the next session adds another one.

## Two detection approaches that look right and are wrong

Sharing these because both would ship a cleanup that kills healthy servers:

- **`ppid == 1` is not a reliable orphan test.** A comment in `server.ts` already warns about this — a legitimate server gets reparented to init during normal startup when the `bun run` wrapper exits or execs. A previous ppid-change check was removed for exactly this reason.
- **Matching the stdin pipe by inode does not work either.** I tried treating "nobody else holds the other end of fd 0" as the orphan test. Checked against four known-healthy processes, it flagged **all four**. Their fd 0 is a `socket:`, not a `pipe:`, and a `socketpair` has a *different inode on each end*, so an inode cross-reference never finds the peer.

What does work as an external test: walk the `ppid` chain looking for a live `claude` ancestor, plus a minimum age to absorb the startup reparenting window.

## Suggested fixes

1. **Don't rely on the event loop to shut down.** Anything that must work when the loop is stuck cannot live on it. A supervisor outside the process, or a hard watchdog that force-exits, would survive the case that actually happens.
2. **Make the PID file a list, or sweep by cwd/argv**, so a new server can reap *all* stale instances rather than only the most recent.
3. **Escalate to `SIGKILL`** when a stale holder does not exit within a short grace period after `SIGTERM`. Today the `SIGTERM` is best-effort and silently does nothing against a spinning process.
4. **Bound the `getUpdates` retry loop** with backoff, and give up after N consecutive 409s instead of spinning forever — a server that cannot own the polling slot has no reason to keep running.
5. **Persist a few lines of the server's own stderr**, so this class of failure is diagnosable after the fact. Right now the only evidence available post-mortem is CPU time.

Happy to gather more detail if it helps — I have a host where this reproduces on its own within days.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.