fix(api): make WatchSandbox loss-aware and resumable
@letv1nnn is already working on this.
Since Sep 15, 2026.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
User Story
As an API or SDK client monitoring a sandbox, I want watch streams to report recoverable loss and resume from a known cursor, so that transient lag or reconnects do not silently lose state or force me to restart observation from scratch.
Problem Statement
SandboxStreamEvent includes a SandboxStreamWarning payload intended for missed-message warnings. The current broadcast receive path converts lag into a RESOURCE_EXHAUSTED stream error and terminates. The request also has no resume cursor, so a reconnect cannot request events after the last successfully processed item.
The separate stop_on_terminal implementation now recognizes ERROR; that corrected behavior needs regression coverage but is not the remaining defect.
Impact / Why This Matters
Slow consumers and transiently disconnected SDK clients lose observability precisely when event volume is high. Polling GetSandbox can recover only the latest sandbox snapshot, not missed log or platform events. Callers cannot distinguish a complete replay from a gap or suppress duplicates reliably after reconnecting.
Proposed Design
Give watch events an ordered resumable position and allow requests to resume after a previously observed cursor. When the server can continue after broadcast lag, emit SandboxStreamWarning with the lost range and continue from the next available item. When history is no longer recoverable, terminate with a documented status and enough structured information for the client to restart safely.
Define cursor scope, retention, ordering across status/log/platform event sources, duplicate behavior, and terminal-phase completion. SDK helpers should reconnect and resume without hiding unrecoverable gaps.
Acceptance Criteria
- Recoverable broadcast lag emits a warning event rather than immediately terminating the stream.
- Every resumable event exposes an ordered cursor or equivalent continuation position.
- A client can reconnect and request events after its last processed cursor.
- Recoverable and unrecoverable gaps have distinct documented behavior.
- Ordering across sandbox snapshots, logs, and platform events is specified.
- SDK helpers expose warnings and support safe resume without silently dropping or duplicating events.
- Tests cover lag, reconnect, cursor expiry, duplicate suppression, terminal READY/COMPLETED/STOPPED/ERROR behavior, and client cancellation.
Reproduction Steps
- Start
WatchSandboxwith one or more live event sources. - Allow the server-side broadcast receiver to lag beyond channel capacity.
- Observe that the gateway converts
RecvError::LaggedtoRESOURCE_EXHAUSTEDand returns, rather than emitting the declared warning payload.
Environment
- OpenShell: source investigation on
mainat69a05ebb3 - OS: not applicable
- Runtime, deployment, or integration: gateway
WatchSandboxbroadcast streams
Alternatives Considered
Terminate on every lag and require clients to poll current state. This cannot recover logs or events and does not use the existing warning contract. Emit warnings without adding cursors. This makes loss visible but still gives reconnecting clients no reliable continuation point.
Agent Investigation
The lag conversion lives in crates/openshell-server/src/sandbox_watch.rs, and the watch producer returns after sending the resulting status. SandboxStreamWarning remains in proto/openshell.proto.
Related: #2565. Source audit: https://gist.github.com/mrunalp/e80942c1544a0225ee588796a41ab30b.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.