anomalyco / anomalyco/opencode

Hung plugin hook silently discards prompts after prompt_async returns 204

Open
#39,031 1 comment 0 reactions 1 assignee View on GitHub

@jlongster is already working on this.

Since Jul 27, 2026.

Dominant language
TypeScript
Stars
209k
Forks
27.5k
PR merge metrics
PR metrics pending

Description

A plugin chat.message hook that never resolves silently discards a prompt that prompt_async already acknowledged with 204. Nothing is logged, no error event is published, and the session is left with its session.created event and no messages. With a real plugin config this was losing 33% of concurrent subagent dispatches on my machine (40/120 at concurrency 12).

Two separate problems combine:

  1. Plugin.trigger runs hooks as yield* Effect.promise(async () => fn(input, output)) (packages/opencode/src/plugin/index.ts). No timeout, and Effect.promise is non-interruptible — a hook whose promise never settles parks the prompt fiber forever.
  2. createUserMessage runs plugin.trigger("chat.message", ...) before sessions.updateMessage(info) (packages/opencode/src/session/prompt.ts), so a hung hook loses a prompt the API already acknowledged.

How I pinned it down: console.error probes along the dispatch path plus an Effect.onExit observer on the forked fiber. Every lost prompt ran normally through part resolution, then parked at plugin.trigger("chat.message"). The exit observer never fired for lost prompts (it logs Success for every healthy one), so the fiber is parked, not failed or interrupted. Per-hook probes inside Plugin.trigger showed a hook-start with no matching hook-done, always for the same hook.

In my config the offender is opencode-discord-presence: its hook awaits a Discord RPC setActivity, and @xhayper/discord-rpc's Client.request() waits on a nonce with no timeout (filed separately: Puri12/opencode-discord-presence#13). Why Discord's reply goes missing under bursts is a guess (presence updates are rate-limited); I didn't packet-trace it. The park itself is directly observed. Which plugin it is doesn't really matter for the core problem though.

Loss rates by plugin config, 12 concurrent prompt_async per trial on an otherwise idle machine:

config lost
discord-presence + oh-my-openagent 40/120
--pure (no external plugins) 0/60
discord-presence alone 6/36
oh-my-openagent alone 0/36

Reproduced on the shipped 1.18.5 binary and a source build of dev (7534d235). To be clear, this does not reproduce plugin-free (--pure is clean). You need a hook that stalls; the script below is just the harness.

With such a plugin loaded:

python repro.py --port <port> --levels 12 --trials 5

Lost sessions have zero message rows. Loss rate rises with concurrency (0% at n=1 across 33 single-dispatch batches, ~33% at n=12 here).

repro.py
"""
Concurrent prompt_async dispatches silently never start.

  POST /session                        -> create session
  POST /session/:id/prompt_async       -> fire-and-forget prompt (returns 204)
  GET  /session/:id/message            -> count persisted messages

A session that never starts ends with ZERO messages. A healthy one has >= 1.
"""
import argparse, json, sys, time, urllib.request, urllib.error, threading, collections

def call(base, method, path, payload=None, timeout=30):
    url = base.rstrip("/") + path
    data = json.dumps(payload).encode() if payload is not None else None
    req = urllib.request.Request(url, data=data, method=method,
                                 headers={"Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = r.read().decode("utf-8", "replace")
            return r.status, (json.loads(body) if body.strip() else None)
    except urllib.error.HTTPError as e:
        return e.code, e.read().decode("utf-8", "replace")[:200]
    except Exception as e:
        return -1, str(e)[:200]

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--port", type=int, required=True)
    ap.add_argument("--host", default="127.0.0.1")
    ap.add_argument("--levels", default="1,2,3,4,6")
    ap.add_argument("--trials", type=int, default=3)
    ap.add_argument("--settle", type=int, default=25, help="seconds to wait before counting")
    ap.add_argument("--model", default=None)
    ap.add_argument("--agent", default=None)
    a = ap.parse_args()

    base = "http://%s:%d" % (a.host, a.port)
    st, _ = call(base, "GET", "/session?limit=1")
    if st not in (200, 204):
        sys.exit("Cannot reach OpenCode at %s (status=%s)" % (base, st))
    print("connected to %s\n" % base)

    levels = [int(x) for x in a.levels.split(",") if x.strip()]
    results = collections.OrderedDict()

    for n in levels:
        empties = total = 0
        for trial in range(a.trials):
            ids = []
            for i in range(n):
                st, body = call(base, "POST", "/session",
                                {"title": "repro-n%d-t%d-%d" % (n, trial, i)})
                if st in (200, 201) and isinstance(body, dict) and body.get("id"):
                    ids.append(body["id"])
                else:
                    print("  session create failed: status=%s body=%.120s" % (st, body))
            if len(ids) != n:
                print("  skipping trial (only %d/%d sessions)" % (len(ids), n)); continue

            barrier = threading.Barrier(n)
            statuses = {}
            def fire(sid):
                payload = {"parts": [{"type": "text", "text":
                            "Reply with exactly the word: ok"}]}
                if a.model:
                    prov, _, mid = a.model.partition("/")
                    payload["model"] = {"providerID": prov, "modelID": mid}
                if a.agent:
                    payload["agent"] = a.agent
                barrier.wait()
                statuses[sid] = call(base, "POST", "/session/%s/prompt_async" % sid, payload)[0]

            threads = [threading.Thread(target=fire, args=(s,)) for s in ids]
            t0 = time.time()
            for t in threads: t.start()
            for t in threads: t.join()
            spread = (time.time() - t0) * 1000

            time.sleep(a.settle)
            empty_ids = []
            for sid in ids:
                st, body = call(base, "GET", "/session/%s/message" % sid)
                cnt = len(body) if isinstance(body, list) else 0
                if cnt == 0: empty_ids.append(sid)
            empties += len(empty_ids); total += len(ids)
            print("  n=%-2d trial%-2d spread=%5.0fms  http=%s  EMPTY %d/%d %s"
                  % (n, trial + 1, spread,
                     ",".join(str(v) for v in statuses.values()),
                     len(empty_ids), len(ids),
                     "<-- REPRODUCED" if empty_ids else ""))
        results[n] = (empties, total)
        print()

    print("%-12s %8s %8s %8s" % ("CONCURRENCY", "EMPTY", "TOTAL", "RATE"))
    for n, (e, t) in results.items():
        print("%-12d %8d %8d %7.0f%%" % (n, e, t, (100.0 * e / t) if t else 0))

if __name__ == "__main__":
    main()

On my production 1.18.5 install: 360 sessions (310 subagent children, 50 roots) created Jun 29–Jul 26 across 1.17.11–1.18.5 with exactly this signature — event_sequence.seq = 1, only session.created.1/session.updated.1 events, no message rows. These are the subagents that "never start".

Suggested fixes:

  • Wrap hook calls in Effect.tryPromise + Effect.timeout and log on timeout/failure, so a stalled hook degrades the plugin instead of the prompt. This alone removes the unbounded silent loss.
  • Persisting the user message before chat.message hooks would close the gap entirely, but hooks can currently mutate the parts pre-persist, so that's a contract change that needs a decision.

Related issues:

  • #38092 — this is a concrete way the 204-before-persistence gap turns into real data loss.
  • #33394 — same user-visible symptom (204, empty child session), different trigger.
  • #31072 — I chased that hypothesis first for the same symptom, but it doesn't hold on current dev: commitDurableEvent runs inside db.transaction(..., { behavior: "immediate" }) and the sqlite client serializes transactions with a 1-permit semaphore (packages/core/src/database/sqlite.bun.ts), and that wrapper predates v1.16.2. I implemented the proposed atomic-upsert fix plus its regression test: the test passes on unpatched code (10 concurrent same-aggregate publishes already get seqs 0–9), and the patch didn't change the loss rate (33% baseline vs 42% patched, i.e. trial noise). Also, 406 of the 412 "orphaned" subagent sessions that seemed to corroborate it on my install turned out to be pre-event-sourcing sessions (v1.2.x–1.16.2) with full message history, not failures.

OpenCode 1.18.5 and dev@7534d235 from source, Windows 11 x64, SQLite WAL.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.