boxlite-ai / boxlite-ai/boxlite
Mid-run shim crash not detected until next operation or runtime restart
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 179
- Avg merge
- 23h 25m
- Merged PRs (30d)
- 121
Description
## Issue
When the shim process crashes during normal operation of a Running box (panic, SIGSEGV, OOM-kill, etc.), the host runtime has no proactive detection. The shim's crash handler writes `~/.boxlite/boxes//exit` with the `ExitInfo` JSON, but nothing on the host reads it until something triggers a later check.
## Current behavior
- DB still reports `status=Running, pid=`
- `BoxImpl.live` (`OnceCell`) holds a `GuestSession` + `ShimHandler` that reference a dead process
- `shim.pid` on disk: PID is dead → `ProcessIdentity::Absent`, but no code path runs to act on it
- `exit` file is on disk, but no host code reads it
## When the crash is finally surfaced
| Trigger | Latency |
|---|---|
| Next RPC against the box (`exec`, `info`, `metrics`, etc.) | until user pokes the box; surfaces as a transport error, not a `Failed` status |
| Runtime restart → `recover_boxes` reads `exit` | until next restart |
| New attach attempt within the same runtime | never (live is cached, no reattach runs) |
So between the crash and the next user operation or restart, the box appears Running while it isn't, and the user sees confusing low-level tonic errors instead of a `Failed` status with the captured `CrashReport`.
## Scope
Tracking only. Detection design is open (background poll vs. reverse watchdog pipe vs. SIGCHLD vs. health-check task hook). Recovery (#TBD), pre-attach preflight, and crash-file consumption are already handled in PR #492 follow-up work; this issue covers the remaining real-time gap.
Contributor guide
Research direction
Start by tracing BoxImpl.live, GuestSession, ShimHandler, and the host paths that inspect shim.pid or consume the exit file; recover_boxes is the existing restart-time entry point. Compare the possible detection approaches named in the issue and define how a mid-run crash should become Failed with the captured CrashReport, without revisiting recovery or pre-attach work from PR #492.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100