BOHICA-LABS / BOHICA-LABS/vsdd-factory
bug(state-manager): transient API connection errors require manual orchestrator retry
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
When `state-manager` (or any sub-agent) fails mid-execution with a transient API error like `Connection closed mid-response`, the failure is reported to the orchestrator as a generic "the response above may be incomplete" string. There is no structured failure signal, no automatic retry, and no way for the orchestrator to distinguish a transient API failure from a permanent agent error.
The orchestrator must manually:
1. Detect the failure (by inspecting the expected output files / STATE.md state).
2. Compose a fresh retry prompt with all the same context.
3. Re-dispatch as a brand-new `Agent` call.
This is error-prone and wastes tokens — the retry prompt has to re-include all the content the failed dispatch already received.
## Reproduction
ftc-blue pass-8 finalization (2026-06-23):
1. Orchestrator dispatched state-manager with two tasks: persist pass-8 report (~150 lines verbatim) and update STATE.md.
2. Agent returned `API Error: Connection closed mid-response. The response above may be incomplete.` after ~345 seconds.
3. Orchestrator inspected `.factory/cycles/cycle-001/adversarial/p1d-pass-8.md` (not created) and STATE.md (unchanged) to confirm failure.
4. Orchestrator composed a retry prompt (~3000 tokens of duplicated context) and dispatched a fresh state-manager.
5. Retry succeeded.
Net cost: ~5k extra tokens + manual recovery overhead.
## Proposed change
Two options:
**Option A — Auto-retry with idempotency.**
The orchestrator's `Agent` tool, or the agent dispatch framework, should auto-retry on `API Error: Connection closed mid-response` (or other transient errors) up to N times (e.g., 3) before surfacing the failure to the orchestrator. Retries must be idempotent — for state-manager especially, this means the prompt should be safe to re-run.
**Option B — Structured failure surface.**
Wrap the agent return value in a structured failure type:
```
{
status: "succeeded" | "failed_transient" | "failed_permanent",
reason?: string,
output: string,
partial?: bool
}
```
The orchestrator can then make informed decisions: retry on `failed_transient`, escalate on `failed_permanent`, audit on `partial`.
Option A is preferred for transient errors; Option B for everything else.
## Applies to
- Agent dispatch framework / `Agent` tool internals
- `agents/state-manager.md` — ensure idempotency of common operations (re-running a write to the same path with the same content should be safe)
- `agents/orchestrator/` — failure-handling protocol
## Acceptance criteria
- [ ] Connection-closed errors auto-retry up to N times before surfacing to caller
- [ ] Orchestrator gets a structured failure signal, not a raw error string in agent output
- [ ] Common state-manager operations are documented as idempotent
- [ ] Documented behavior for non-transient failures (e.g., agent died on logic error)
## Found during
ftc-blue Phase 1d pass-8 finalization (2026-06-23, vsdd-factory@1.0.0-rc.21).
## Notes
Transient API errors are inevitable in any system that talks to upstream LLMs. The fact that this manifests as a manual-recovery exercise for the orchestrator suggests the failure-handling design lags behind the actual error model.
Contributor guide
Assessment
This issue has not been assessed yet.