[Epic] Streaming: separate the wire format from the durable event record

The problem

Streamed text leaves the agent on two channels that know nothing about each other, and only one of them is tracked.

Durable events go through the callback chain and get written to disk. Token deltas are handed straight to PubSub from whichever thread produced them. Nothing orders one against the other, and nothing links them — so the browser puts them back together by comparing strings.

flowchart LR
    U["user callbacks<br/>(incl. the publish wrapper)<br/><b>runs 1st</b>"] --> DC["_default_callback<br/><b>append_event — persists — 2nd</b>"]
    U -.-> PS
    DC --> DISK[("event-{idx:05d}-{id}.json")]
    TOK["on_token(chunk)"] --> PS
    ACP["ACPAgent bridge<br/>(bypasses llm.py)"] --> TOK
    PS["PubSub — no kind filter"]
    PS --> WS["WebSocket"] --> BR["Browser<br/>re-marries by string matching"]
    PS --> WH["Webhook"]
    PS --> TL["Telemetry"]

The root cause is one decision: our socket frame is our disk record.

• sockets.py:489 sends event.model_dump(mode="json", exclude_none=True)
• event_store.py:217 persists event.model_dump_json(exclude_none=True)
• remote_conversation.py:239 decodes with Event.model_validate

Same class, same serialization, same validator — and Event is extra="forbid", so every wire change is a storage migration and every storage change is a wire break. That is why deltas carry no identity: there was nowhere to put it that didn't cost a migration.

The whole problem in one sentence

StreamingDeltaEvent carries text, a random id and a timestamp — and nothing else. No stream identity, no sequence, no attempt number, no pointer to the message that supersedes it.

So the client can't tell a re-streamed retry from new text, can't spot a gap, can't place a delta against a message that arrived beside it, and can't tell when a stream ended. It answers all four by guessing at the text. That is what the 448 lines of handle-event-for-ui.ts are: protocol repair in the view layer.

Nobody else ships a delta this way

Checked against the installed packages, not the documentation.

	What a delta says about itself	How a message starts and ends
pi	messageId · contentIndex · kind	item_started → item_updated → item_finished
OpenAI Responses	item_id · output_index · content_index · sequence_number	output_item.added → .done
Anthropic Messages	index	content_block_start → _stop
ACP	nothing	agent_message_chunk, in arrival order
OpenHands	nothing	nothing — and on a second, unordered channel

And ours is not missing upstream — we drop it. _token_streaming_callback (event_service.py:1063-1075) reads delta.content and delta.reasoning_content and nothing else. chunk.id and choice.index are in scope on that line and never read; _publish_stream_delta has no parameter for them. litellm hands us the identity intact and we discard it.

What it breaks

Issue	What happens	Missing
OpenHands/OpenHands#15433	A user message lands mid-stream; the next delta no longer sees a delta last and opens a second bubble. On finalize only the later half is superseded — the first half survives next to the finished message	A turn-scoped slot
OpenHands/OpenHands#15432	One FIFOLock guards the event log, agent, execution status, secrets, stats and HEAD	Separate locks
#4077	Deltas fire inside the retry boundary, so a retry re-streams everything; delta futures are discarded; the resume cursor compares naive local timestamps	An attempt number, a tracked path, a real cursor
OpenHands/OpenHands#15720	ACP masks each chunk alone, so a secret split across chunks matches neither in the deltas	One boundary both paths cross
OpenHands/OpenHands#15493	Text is committed at whatever granularity the network delivered it	A render clock
OpenHands/OpenHands#15511	The compose form forces stream: true and the server re-decides	One owner for the flag
OpenHands/OpenHands#16331	Conversation-scoped requests queue behind a process-wide lock	Per-conversation locking
#4672	Every delta is POSTed to your webhooks	A kind filter
#4673	Telemetry counts deltas as conversation activity	The same

Those last two are one bug, and nobody chose it. Telemetry and webhooks both register through event_service.subscribe_to_events → self._pub_sub.subscribe(...) — the same PubSub that _publish_stream_delta publishes to — and PubSub.__call__ (pub_sub.py:81-102) has no kind filter. The telling detail: StreamingDeltaEvent appears in the agent-server exactly three times — an import (event_service.py:66), a comment (:1048) and its construction site (:1056). No isinstance, no filter, nowhere. Nobody decided to include deltas; nobody had a place to exclude them. That is what making a delta an Event costs — it opts into every consumer of the event bus by default.

Update. #4689 fixed both by making delivery opt-in (Subscriber.receives_streaming_deltas), and in doing so surfaced the third consumer this paragraph predicted — but inverted. _EventSubscriber (conversation_service.py:2433) did not forget to opt out; the standard streaming path had quietly come to depend on it, because it calls update_last_execution_time() and that is what keeps the runtime-api from reaping the pod. It does not opt in, so that heartbeat is now gone: #4695. Two consumers forgot to opt out, and a third came to depend on it. Nobody chose either. That is the cost, stated more precisely than the original claim: a shared bus does not just leak traffic outward, it grows load-bearing dependencies inward that no one declared.

Desired Behavior

Stop shipping the disk record over the socket; then give the stream an identity it can be closed by.

1 · The wire stops being the disk. A new endpoint, /sockets/session/{id}, speaking an envelope that is not an Event. Event doesn't change at all — same class, same extra="forbid", same bytes on disk. It rides inside the envelope as a payload. The old endpoint is frozen as-is; a client speaks one or the other, so the URL is the protocol version — no handshake, no protocol=2 field, no dual-decode path.

The sequence number needs no migration: files are already event-{idx:05d}-{event_id}.json and EventLog.append has self._length in hand at write time (event_store.py:222-227). It just has to return it.

2 · The stream's identity is minted when it opens. StreamContext.open() allocates the event id up front, and the message is built with it:

ctx = StreamContext.open(state)    # mints item_id
...                                 # Delta frames reference it
MessageEvent(id=ctx.item_id, …)     # the message IS the item

This is safe against the append-only log because minting is not a write. Event.id is already client-minted in-process (base.py:24-26, default_factory=lambda: str(uuid.uuid4())). The log never assigns ids. So this changes only when uuid4() runs. Same bytes, one append, same ordering. If the stream dies the id is simply never used.

A client holding an open slot for item_id sees the durable frame arrive with event.id == item_id and swaps the slot for the real message. That is the entire close protocol — one equality test, on a field that already exists. Gone with it: the close frame, the watermark, the final-order counter, and every line of text comparison in the browser.

3 · Bound the connection in bytes, not in queues. A connection that can't keep up is dropped, not buffered — safe because the durable side has a cursor, so the client reconnects with after_seq=N and loses nothing. Disconnection is the backpressure mechanism.

The protocol

Frame	Carries	When
Durable	seq, event	Once per durable event, after it's on disk. event is the existing JSON, unchanged
ItemStarted	item_id, attempt, anchor_seq	Once per attempt, before the first token. A higher attempt supersedes a lower one
Delta	item_id, attempt, order, kind, content	Per masked chunk. kind is text or reasoning
ItemAborted	item_id, attempt, reason	Only when a stream ends without producing a message

Rules: Durable never goes missing across a reconnect (within a connection it's recoverable via the cursor). Delta may be dropped freely. Every ItemStarted is retired by exactly one Durable or one ItemAborted — including on cancellation, provider failure, policy rejection and unhandled exception. Progress frames are never replayed. No ordering is promised between two open items.

Plan

Step	Work	Issue	Needs
0	Reproductions and baselines	#4679	—
1	Independent repo fixes	#4674 · #4675 · #4676 · #4677 · #4678 · #4077	—
1b	Deltas get their own fan-out: StreamingDeltaEvent stops subclassing Event; delta-only PubSub; idle heartbeat carried across	#4696 · #4695	—
2	append_event returns its sequence number; publish moves after persist	#4680	—
3	New endpoint + envelope, durable events only; cursor, paged replay, atomic reconnect boundary	#4681	2
4	Byte-budget admission, one writer per connection	#4681	3, 0
5	StreamContext mints the id; progress frames across all four entry points	#4682	3
6	Canvas turn model	OpenHands/OpenHands#16965	5
7	Delete StreamingDeltaEvent and its bus; freeze the old endpoint	#4683	6, 1b

Client endpoint switch (part of step 3). Three sites, two of them now in this repo since OpenHands/typescript-client was archived and moved to clients/typescript:
remote_conversation.py · clients/typescript (#4763) · canvas (OpenHands/OpenHands#16965)

The TypeScript client no longer has its own release train — it publishes off this repo's v* release and tracks the SDK version. But it builds and integration-tests against the agent-server release pinned in clients/typescript/package.json (config.agentServerImage), currently 1.44.0, which predates #4807. That pin has to be bumped to a release containing the new endpoint before #4763 can start.

Closed by #4689, which made delta delivery opt-in rather than opt-out:
#4672 (deltas POSTed to webhooks) · #4673 (telemetry counts deltas)

That fix is a filter, and it is deliberately interim — its own description says it "does not preclude the epic's structural direction of taking deltas off the bus entirely." Step 1b is that structural half, and it removes the flag #4689 added. It also has to restore the heartbeat that fix dropped (#4695).

Publish-after-persist (step 2), stated precisely — because it sounds scarier than it is. Nothing partial is ever written and no event reaches disk before it's finished; this reorders two existing callbacks over an already-complete event. Today composed_list = callback_list + [_default_callback] (local_conversation.py:427) puts the user callbacks — including the one that publishes — ahead of _default_callback, which calls state.append_event(e) (:418), and compose_callbacks runs them in list order (base.py:473-476). So the socket is told before the disk is written, and if the append then raises we have already announced an event that does not exist. Swap them. The log ends up taking fewer writes than today, since deltas are never appended at all.

Six tests that don't exist

The current streaming test covers deltas being emitted and suppressed, and nothing else. Missing: retry × identity · reconnect × slot discard · replay × live interleave · ACP × split-chunk secret · error-after-tokens × exactly one retire · wedged connection × publisher never blocks. All six fail today.

Not modelled yet

Sub-agent streams sharing the channel (#3907) · StreamContext across a mid-turn model switch · whether the single-pass masker compiles cheaply enough for conversations with many secrets.

---

Read against software-agent-sdk @ a1a2cdb (rel-1.43.0) and OpenHands @ a80b1bb. Every code reference above was verified against the checkout.

Acceptance Criteria

• Step 0 baselines landed (#4679) — the byte budget is derived from a measured frame-size distribution, not a guess.
• Step 1 independent fixes landed: #4674 · #4675 · #4676 · #4677 · #4678 · #4077.
• Step 1b landed (#4696) — StreamingDeltaEvent is not an Event, deltas have their own fan-out, the frame on the wire is unchanged, and receives_streaming_deltas is gone.
• The runtime idle timer is reset during a long stream that produces no durable events (#4695), and stays so after step 1b.
• Step 2 landed (#4680) — append_event returns its sequence number and publish happens after persist.
• Steps 3+4 landed (#4681) — /sockets/session/{id} serves durable events in a non-Event envelope, with cursor, paged replay and byte-budget admission.
• Step 5 landed (#4682) — StreamContext mints the identity; every ItemStarted is retired by exactly one Durable or ItemAborted.
• Step 6 landed (OpenHands/OpenHands#16965) — canvas keys slots by event id and handle-event-for-ui.ts contains no text comparison.
• Step 7 landed (#4683) — StreamingDeltaEvent is deleted outright and the old endpoint is documented as deprecated.
• Event is unchanged on disk throughout: same class, same extra="forbid", same bytes.