BOHICA-LABS / BOHICA-LABS/vsdd-factory

bug(state-manager): transient API connection errors require manual orchestrator retry

Open
#212 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

When `state-manager` (or any sub-agent) fails mid-execution with a transient API error like `Connection closed mid-response`, the failure is reported to the orchestrator as a generic "the response above may be incomplete" string. There is no structured failure signal, no automatic retry, and no way for the orchestrator to distinguish a transient API failure from a permanent agent error.

The orchestrator must manually:

1. Detect the failure (by inspecting the expected output files / STATE.md state).
2. Compose a fresh retry prompt with all the same context.
3. Re-dispatch as a brand-new `Agent` call.

This is error-prone and wastes tokens — the retry prompt has to re-include all the content the failed dispatch already received.

## Reproduction

ftc-blue pass-8 finalization (2026-06-23):

1. Orchestrator dispatched state-manager with two tasks: persist pass-8 report (~150 lines verbatim) and update STATE.md.
2. Agent returned `API Error: Connection closed mid-response. The response above may be incomplete.` after ~345 seconds.
3. Orchestrator inspected `.factory/cycles/cycle-001/adversarial/p1d-pass-8.md` (not created) and STATE.md (unchanged) to confirm failure.
4. Orchestrator composed a retry prompt (~3000 tokens of duplicated context) and dispatched a fresh state-manager.
5. Retry succeeded.

Net cost: ~5k extra tokens + manual recovery overhead.

## Proposed change

Two options:

**Option A — Auto-retry with idempotency.**

The orchestrator's `Agent` tool, or the agent dispatch framework, should auto-retry on `API Error: Connection closed mid-response` (or other transient errors) up to N times (e.g., 3) before surfacing the failure to the orchestrator. Retries must be idempotent — for state-manager especially, this means the prompt should be safe to re-run.

**Option B — Structured failure surface.**

Wrap the agent return value in a structured failure type:

```
{
status: "succeeded" | "failed_transient" | "failed_permanent",
reason?: string,
output: string,
partial?: bool
}
```

The orchestrator can then make informed decisions: retry on `failed_transient`, escalate on `failed_permanent`, audit on `partial`.

Option A is preferred for transient errors; Option B for everything else.

## Applies to

- Agent dispatch framework / `Agent` tool internals
- `agents/state-manager.md` — ensure idempotency of common operations (re-running a write to the same path with the same content should be safe)
- `agents/orchestrator/` — failure-handling protocol

## Acceptance criteria

- [ ] Connection-closed errors auto-retry up to N times before surfacing to caller
- [ ] Orchestrator gets a structured failure signal, not a raw error string in agent output
- [ ] Common state-manager operations are documented as idempotent
- [ ] Documented behavior for non-transient failures (e.g., agent died on logic error)

## Found during

ftc-blue Phase 1d pass-8 finalization (2026-06-23, vsdd-factory@1.0.0-rc.21).

## Notes

Transient API errors are inevitable in any system that talks to upstream LLMs. The fact that this manifests as a manual-recovery exercise for the orchestrator suggests the failure-handling design lags behind the actual error model.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.