fix(api): make WatchSandbox loss-aware and resumable
@letv1nnn ya está trabajando en esto.
Desde el 15/9/2026.
- Lenguaje dominante
- Rust
- Estrellas
- 8.7k
- Forks
- 1.3k
- Merge medio
- 2 d 11 h
- PR fusionados (30 d)
- 253
Descripción
User Story
As an API or SDK client monitoring a sandbox, I want watch streams to report recoverable loss and resume from a known cursor, so that transient lag or reconnects do not silently lose state or force me to restart observation from scratch.
Problem Statement
SandboxStreamEvent includes a SandboxStreamWarning payload intended for missed-message warnings. The current broadcast receive path converts lag into a RESOURCE_EXHAUSTED stream error and terminates. The request also has no resume cursor, so a reconnect cannot request events after the last successfully processed item.
The separate stop_on_terminal implementation now recognizes ERROR; that corrected behavior needs regression coverage but is not the remaining defect.
Impact / Why This Matters
Slow consumers and transiently disconnected SDK clients lose observability precisely when event volume is high. Polling GetSandbox can recover only the latest sandbox snapshot, not missed log or platform events. Callers cannot distinguish a complete replay from a gap or suppress duplicates reliably after reconnecting.
Proposed Design
Give watch events an ordered resumable position and allow requests to resume after a previously observed cursor. When the server can continue after broadcast lag, emit SandboxStreamWarning with the lost range and continue from the next available item. When history is no longer recoverable, terminate with a documented status and enough structured information for the client to restart safely.
Define cursor scope, retention, ordering across status/log/platform event sources, duplicate behavior, and terminal-phase completion. SDK helpers should reconnect and resume without hiding unrecoverable gaps.
Acceptance Criteria
- Recoverable broadcast lag emits a warning event rather than immediately terminating the stream.
- Every resumable event exposes an ordered cursor or equivalent continuation position.
- A client can reconnect and request events after its last processed cursor.
- Recoverable and unrecoverable gaps have distinct documented behavior.
- Ordering across sandbox snapshots, logs, and platform events is specified.
- SDK helpers expose warnings and support safe resume without silently dropping or duplicating events.
- Tests cover lag, reconnect, cursor expiry, duplicate suppression, terminal READY/COMPLETED/STOPPED/ERROR behavior, and client cancellation.
Reproduction Steps
- Start
WatchSandboxwith one or more live event sources. - Allow the server-side broadcast receiver to lag beyond channel capacity.
- Observe that the gateway converts
RecvError::LaggedtoRESOURCE_EXHAUSTEDand returns, rather than emitting the declared warning payload.
Environment
- OpenShell: source investigation on
mainat69a05ebb3 - OS: not applicable
- Runtime, deployment, or integration: gateway
WatchSandboxbroadcast streams
Alternatives Considered
Terminate on every lag and require clients to poll current state. This cannot recover logs or events and does not use the existing warning contract. Emit warnings without adding cursors. This makes loss visible but still gives reconnecting clients no reliable continuation point.
Agent Investigation
The lag conversion lives in crates/openshell-server/src/sandbox_watch.rs, and the watch producer returns after sending the resulting status. SandboxStreamWarning remains in proto/openshell.proto.
Related: #2565. Source audit: https://gist.github.com/mrunalp/e80942c1544a0225ee588796a41ab30b.
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Evaluación
Este issue todavía no se ha evaluado.