galaxyproject / galaxyproject/loom

Orbit: resume-hardening after laptop suspend / long idle (stale Galaxy state, dead-socket tool failures)

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14
Forks
12
Avg merge
6d 5h
Merged PRs (30d)
17

Description

## Symptom

Real-world laptop usage: open Orbit, kick off a project (Galaxy invocations queued, local downloads running), close the lid, go home, reopen the laptop hours later. Most things actually survive — kernel `SIGSTOP`s/`SIGCONT`s local processes across suspend, Galaxy work runs server-side regardless, detached subprocesses keep going. But two failure modes leak through:

1. **Stale Galaxy state in the brain.** Active invocations may have completed (or failed) during the suspend window. Brain still trusts whatever it cached pre-suspend — UI shows "queued" or "running" for a job that finished hours ago, until something forces a refresh.

2. **Dead-socket tool failures pass silently.** Any TCP connection open at suspend time (curl mid-download, an Anthropic stream, a Galaxy poll) is timed out by the server / NAT during the suspend gap. On resume the socket is dead. The tool returns an error, but the brain doesn't always recognize this as "transient — retry" vs "real failure" — so a single laptop close can permanently sink a download URL or a polling loop.

## Why

Suspend/resume is invisible to the brain. There's no `system_resumed` event, no "you've been idle for N hours" hook. Everything just continues from the last in-memory state, which assumes connections that no longer exist.

## Proposed fix

### Resume detection

Cheap heuristic: at every `turn_start`, check `wallClockNow - lastEventTimestamp`. If gap > threshold (5 min? 30 min?), flag the turn as `resumed_after_gap` and trigger a state refresh before proceeding. No OS-level suspend hook needed; the gap itself is the signal.

```ts
// in turn_start handler
const gap = Date.now() - state.lastActivityAt;
if (gap > RESUME_THRESHOLD_MS) {
await refreshGalaxyState(ctx);
}
state.lastActivityAt = Date.now();
```

### Galaxy state refresh

On detected resume:
- Re-fetch the current history's invocations (status, progress).
- Reconcile cached invocation state with truth from the server. Update notebook / status bar accordingly.
- If the user had an active `/run` or watchdog poll, re-arm it.

### Tool failure classification

Tool wrappers (bash, galaxy invocation poller, anthropic API client) need to distinguish:
- **Transient network failure** (`ECONNRESET`, `ETIMEDOUT`, HTTP 5xx, curl exit codes 7/28/35/56) → mark retryable, surface to brain with `retryable: true`.
- **Real failure** (HTTP 401/403/404, file-not-found, malformed input) → surface as terminal.

Brain side: when it sees `retryable: true`, decide based on context — for a download script, retry the failing URL; for a polling loop, just re-arm; for an Anthropic stream interrupted mid-turn, replay the last user message silently.

## Files

- `extensions/loom/index.ts` — `turn_start` hook that checks the gap and triggers refresh; needs a `lastActivityAt` field in session state.
- `extensions/loom/galaxy/*` — a `refreshActiveInvocations(ctx)` helper that fetches and reconciles.
- Bash tool wrapper (extensions/loom) — exit-code → `retryable` mapping for known transient codes.
- Anthropic client wrapper — wrap stream errors, classify by error type.

## Edge cases

- User legitimately walked away for an hour without closing the laptop. Same gap-based heuristic fires; refresh happens; cost is one extra Galaxy API call. Acceptable.
- Suspend during the *very first* turn (no prior `lastActivityAt`): just skip the refresh; nothing to reconcile yet.
- Galaxy server itself was down during suspend: refresh fails. Don't crash the turn — surface a single "couldn't refresh Galaxy state, working from cache" warning and proceed.
- Brain crashed and was relaunched (vs suspended): looks identical to a long gap; refresh handles it the same way. Bonus: reconciles cleanly without a separate "post-restart" code path.

## Out of scope

Not handling the case where the brain process itself died during suspend — that's a brain restart, already covered by session-bootstrap. This issue is purely about *the brain stayed alive but the world moved on*.

Detached-process visibility post-resume is covered by #70.
Liveness indicator during long turns is covered by #71.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.