boxlite-ai / boxlite-ai/boxlite

Mid-run shim crash not detected until next operation or runtime restart

Open
#586 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.3k
Forks
179
Avg merge
23h 25m
Merged PRs (30d)
121

Description

## Issue

When the shim process crashes during normal operation of a Running box (panic, SIGSEGV, OOM-kill, etc.), the host runtime has no proactive detection. The shim's crash handler writes `~/.boxlite/boxes//exit` with the `ExitInfo` JSON, but nothing on the host reads it until something triggers a later check.

## Current behavior

- DB still reports `status=Running, pid=`
- `BoxImpl.live` (`OnceCell`) holds a `GuestSession` + `ShimHandler` that reference a dead process
- `shim.pid` on disk: PID is dead → `ProcessIdentity::Absent`, but no code path runs to act on it
- `exit` file is on disk, but no host code reads it

## When the crash is finally surfaced

| Trigger | Latency |
|---|---|
| Next RPC against the box (`exec`, `info`, `metrics`, etc.) | until user pokes the box; surfaces as a transport error, not a `Failed` status |
| Runtime restart → `recover_boxes` reads `exit` | until next restart |
| New attach attempt within the same runtime | never (live is cached, no reattach runs) |

So between the crash and the next user operation or restart, the box appears Running while it isn't, and the user sees confusing low-level tonic errors instead of a `Failed` status with the captured `CrashReport`.

## Scope

Tracking only. Detection design is open (background poll vs. reverse watchdog pipe vs. SIGCHLD vs. health-check task hook). Recovery (#TBD), pre-attach preflight, and crash-file consumption are already handled in PR #492 follow-up work; this issue covers the remaining real-time gap.

Contributor guide

Open the contributing guide

Research direction

Start by tracing BoxImpl.live, GuestSession, ShimHandler, and the host paths that inspect shim.pid or consume the exit file; recover_boxes is the existing restart-time entry point. Compare the possible detection approaches named in the issue and define how a mid-run crash should become Failed with the captured CrashReport, without revisiting recovery or pre-attach work from PR #492.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.