boxlite-ai / boxlite-ai/boxlite

GUEST_READY_TIMEOUT is a hard-coded 30s with no override

Open
#1,111 0 comments 0 reactions 0 assignees View on GitHub
bug rust
Dominant language
Rust
Stars
2.3k
Forks
179
Avg merge
23h 25m
Merged PRs (30d)
121

Description

## Summary

`GUEST_READY_TIMEOUT` is a hard-coded 30 s with no environment or config override, so a box
that boots slowly on a contended host fails permanently with no available mitigation:

```
Box uPKTB5hOHtZf failed to start: timeout after 30s

Evidence at T+30s:
• shim_alive = true
• console_bytes = 0
• ready_socket_exists = true
• likely_cause = guest agent never wrote to console (init or vsock plumbing broken)
```

`src/boxlite/src/litebox/init/tasks/guest_connect.rs:110`:

```rust
const GUEST_READY_TIMEOUT: Duration = Duration::from_secs(30);
```

It has exactly two references — the definition and the call site at `:85`. There is no env var,
no `AdvancedOptions` field, and no per-box knob, so a caller on a loaded machine has no way to
raise it. The doc comment says it is "exposed as a constant so tests can call
`wait_for_guest_ready` with a short timeout", i.e. tests can shorten it but production callers
cannot lengthen it.

## How it showed up

Running the full Python SDK integration suite on macOS (`make test:integration:python`, which
the pre-push hook runs), `tests/test_tcp_filter.py::TestEdgeCases::test_multiple_allowed_hosts`
intermittently fails with the message above. It failed 3 times during full-suite runs.

## Reproduction attempts — what does and does not trigger it

I could not reproduce it in isolation. Recording the negative results so the next person does
not repeat them:

| Condition | Result |
|---|---|
| Standalone, idle machine (3 consecutive runs) | pass — 15.9 s, 15.6 s, 17.6 s |
| Standalone, 24 CPU spinners (load avg 20.6 / 18 cores) | pass — 28.4 s |
| Standalone, 40 CPU spinners + 4 `dd` I/O loops (load avg 52.9) | pass — 69.2 s |
| Whole `test_tcp_filter.py` file, 21 boxes back to back | pass — 21/21, 179 s |
| During / immediately after the **full** integration suite | **fails**, 3 occurrences |

Two things worth noting:

- Wall time degrades sharply with host load (15.9 s idle → 28.4 s → 69.2 s), so box startup is
clearly contention-sensitive; but pure CPU/IO pressure alone did not push the *guest-ready
handshake* past 30 s here.
- The failure signature is `console_bytes = 0` — the guest wrote *nothing*, which does not look
like ordinary slowness. Something about running the whole suite (hundreds of boxes) is the
trigger; I did not isolate what.

## Why the constant is worth fixing regardless

Whatever the underlying trigger turns out to be, a user or CI job that hits it today has no
escape hatch: they cannot raise the deadline to confirm "slow" versus "hung", and the same
30 s applies to a warm laptop and a heavily loaded CI box alike.

Suggested: make it overridable (env var and/or an `AdvancedOptions` field) with the current
30 s as the default, keeping the existing diagnostic block on expiry.

## Related

- #469 — the Node SDK hits the same class of failure under parallel test execution. That one is
a *vitest* 120 s hook timeout, i.e. a test-framework limit; this issue is about boxlite's own
internal deadline, which no caller can configure.

## Environment

- macOS 27.0.0, arm64, 18 cores
- boxlite 0.9.7, local (non-REST) runtime, `sdks/python` built from source

Contributor guide

Open the contributing guide

Research direction

Start in src/boxlite/src/litebox/init/tasks/guest_connect.rs, reading the GUEST_READY_TIMEOUT definition and its call site around lines 85 and 110. Trace how AdvancedOptions and per-box configuration are passed, then run the relevant integration command; done means callers can raise the timeout while 30 seconds remains the default and expiry diagnostics are preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.