planetarium / planetarium/vicoop-codex-cli

serve: model self-heal retries are invisible (inflate caller's time-to-first-byte with no attribution)

Open
#22 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2
Forks
1
Avg merge
10m
Merged PRs (30d)
4

Description

Context

serve's postUpstreamWithHeal (src/commands/serve.ts) retries the upstream call up to MAX_MODEL_HEALS (3) times when the auto-resolved default model is rejected — each retry is a full upstream POST (postUpstream) plus a /models re-resolve. The HTTP 200 to the caller is only written after this loop resolves, so every hidden retry round-trip is folded into the caller's time-to-first-byte.

The retries are effectively silent: markModelRejected / resolveDefaultModel don't emit a per-attempt stderr line, and the only existing logs are logError on a final failure.

Why it matters

The bridge client (@vicoop-bridge/client, vicoop-codex backend) now emits a per-task timing breadcrumb (planetarium/vicoop-bridge#370) that splits a turn into serveReady → firstByte → firstDelta → total. That split fully covers the normal pass-through path. The one thing it can't attribute is a self-heal storm: when heals fire, the client just sees an inflated firstByteMs and can't tell "slow connection" from "N hidden upstream round-trips swapping retired models."

Proposal

Make heal attempts observable from the operator's side. Minimal options:

  1. serve-side log — one stderr line per heal in postUpstreamWithHeal: rejected model, the healed-to model, attempt index, and elapsed for that attempt. (Note: the bridge client currently captures the serve child's stderr into a 16 KB ring buffer used only for spawn-failure diagnostics, so to actually surface this we'd also need the client to forward serve stderr at debug — track that as the bridge-side half.)
  2. response signal — surface heal count (e.g. a response header or a field on the terminal usage/metadata) so the bridge backend can fold a healsN into its existing timing line. This keeps it on the structured path and avoids the stderr-forwarding problem entirely. Probably the cleaner of the two.

Priority

Low / deferred. Self-heal only triggers on model-rejected errors (rare), and it's a different failure mode than the slow-turn investigation that motivated the bridge-side timing. File-and-forget until we actually hit an unexplained large firstByteMs in the wild.

Ref: planetarium/vicoop-bridge#370

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/commands/serve.ts at postUpstreamWithHeal, then read postUpstream, markModelRejected, and resolveDefaultModel to trace each retry and model re-resolution. Compare the proposed serve-side stderr signal with the response signal, noting the bridge-side reference and its stderr ring buffer. Done means heal attempts are observable with enough attribution to distinguish them from ordinary first-byte latency.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.