stacklok / stacklok/mecatl

HTTP SSE run streams: emit periodic keepalive comments during silent work

Open
#396 0 comments 0 reactions 1 assignee View on GitHub

@jhrozek is already working on this.

Since Aug 5, 2026.

bug
Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Split out from #391 (claim 2).

Claim

relayRunSSE flushes the initial headers but emits no bytes until a session event is available. A long provider, router-classifier, or MCP call can leave the connection silent long enough for browsers or intermediaries to treat it as dead. Periodic SSE comments (: keepalive\n\n) would preserve the connection without changing the event contract.

Investigation (verified against the codebase)

Verdict: confirmed in full — every SSE path lacks a heartbeat.

Evidence

All four SSE endpoints block with zero byte flow after the header flush; none has a ticker/select:

  • relayRunSSE (internal/adapter/server/http.go:474-536) — shared by both run-entry points: bare for ev := range run.Events() at http.go:511 after the flush at :481.
  • relayEventsSSE (approve-plan merged stream): bare range at http.go:682.
  • runTeam (http.go:929-994): sink-closure writeFrame at :948-978, no background writer.
  • streamSessionEvents (replay): bare range at http.go:1370 (lower severity — disk-bound replay gaps are short).

The gRPC comparison cuts the opposite way one might expect: servers are built with no keepalive options (cmd/mecated/main.go:1464-1471, cmd/mecak8s/serve.go:59-66), but grpc-go clients send HTTP/2 PINGs by default and server default enforcement permits them — so gRPC streams have transport liveness while raw SSE has nothing. The browser/intermediary framing lands precisely on the SSE surface only.

Severity reality-check

Streaming turns are not silent (message deltas, EvToolCall cards, EvNoProgress, EvCompaction). The true silent windows:

  1. Pre-first-chunk provider establishment — bounded by PerAttemptTimeout (300s);
  2. Mid-stream LLM stalls — bounded by StreamIdleTimeout (180s);
  3. Slow non-streaming tool calls (MCP/WebFetch/Bash) — UNBOUNDED by llmresilience (it only watches LLM streams) — the strongest part of the claim;
  4. Awaiting-approval parks.

Browsers/EventSource have no idle timeout themselves; the killers are intermediaries: nginx proxy_read_timeout 60s default, AWS ALB idle 60s default, Cloudflare ~100s. Windows 1–3 all exceed 60s-class proxies today.

Client-parser risk is nil: mecatui is gRPC-only (never parses SSE), ACP is stdio JSON-RPC, and : keepalive comment lines are spec-ignored by EventSource. In-repo precedent: internal/adapter/openaichat/ssefilter.go:66-196 exists precisely to strip inbound : ping comment keepalives.

Proposed fix (~60-80 LoC + tests, ~½ day)

  • Convert the range in relayRunSSE (and relayEventsSSE, same shape) to select { case ev := <-run.Events(): …; case <-ticker.C: … }; on tick, if not failed, write : keepalive\n\n directly to w (never through relayEvent/appendEvent — heartbeats are not session events and must never touch the durable log), flush, and treat a write error as fail() exactly like a data-frame failure.
  • runTeam needs a mutex-guarded writer or is reasonably deferred from v1 (team rounds emit member events regularly); streamSessionEvents skip in v1.
  • Interval 25-30s (under the 60s proxy floor) as a server.Config field + --sse-keepalive-interval flag (0 disables) threaded through both composition roots, mirroring the --llm-stream-idle-timeout idiom.
Tests
  • relaydrain_test.go — assert ticks stop after the first write error (TestSSEWriteErrorDrainsBusyRun asserts exactly-one-Write on a failing writer).
  • http_test.go — blocked mockllm + shrunken interval, asserting a comment arrives before the first event without breaking JSON framing.

Constraints

  • Drain-to-discard sticky failed flag must gate ticks (the AGENTS.md relay invariant).
  • Heartbeats bypass the durable event log entirely.
  • api-compat untouched (server-only change); no proto change.
  • docs/usage/http-sse-api.md gains a one-line note → task docs/task generate for llms.txt.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.