code-yeongyu / code-yeongyu/senpi

Codex WebSocket failures lose partial output and diagnostics through credential rotation; block-start events suppress recovery

Open
#1,628 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
429
Forks
98
Avg merge
5h 3m
Merged PRs (30d)
526

Description

## Environment

- macOS Darwin 25.6.0, arm64.
- Bun 1.4.2.
- omo-ai 5.0.0-0.beta.55, @code-yeongyu/senpi 2026.9.11.
- Provider/model in the inspected failures: openai-codex / gpt-6-astra.
- Both installed OMO copies currently report those versions. Older running processes may have loaded earlier code; the versions of every historical failure are not independently established.

## Observed symptom

`Error: senpi:no-turn-retry:WebSocket closed 1006 Connection ended`

The shared session log contains 11 provider_error entries with `WebSocket closed 1006 Connection ended` between 2026-09-10 00:00 UTC and 2026-09-12 10:14:21 UTC. Eight carry the suppression prefix. These are log-entry counts, not a measured failure rate or necessarily 11 distinct user turns.

Separately, inspection of 95 main-session JSONL files whose names start in September found five exact prefixed 1006 assistant errors across four sessions. All five persisted `content: []`, zero usage, and no diagnostics.

The empty final message does NOT establish that nothing was emitted. The reproduction below shows that the wrapper discards this information.

## Deterministic local reproduction

Save the self-contained script below as `reproduce.mjs` and run it against the installed package:

```sh
bun reproduce.mjs /path/to/node_modules/@code-yeongyu/senpi
```

It uses the shipped `lazyStream`, `streamWithCredentialRotation`, and `classifyCredentialFailure` with an in-memory credential repository and a synthetic provider stream. It makes no network request, reads no credentials, and does not modify installed code.

Observed:

| Provider event sequence before error | Final error prefix | Attempts | Original content/usage/diagnostics preserved |
| --- | --- | --- | --- |
| start | No | 1 | No |
| start, thinking_start (no delta) | Yes | 1 | No |
| start, text_delta | Yes | 1 | No |

The classifier returns `fail_request` for `WebSocket closed 1006 Connection ended`, but `retry_same` with maxAttempts 2 for `ECONNRESET`.

This reproduces local error-handling behavior, not the physical cause of the remote disconnect.

## Relevant code

Pinned version: `0a22329afff58b0c8c548f66fb4f51d358316c89` (v2026.9.11).

- `packages/coding-agent/src/core/credential-pool/rotation-stream.ts`: `errorFromEvent` converts the original assistant error to `new Error(message)`. `isCommittedOutput` is `event.type !== "start"`, so a `thinking_start` event without a delta already prevents replay.
- `packages/coding-agent/src/core/credential-pool/failover.ts`: adds the suppression prefix after committed output.
- `packages/ai/src/api/lazy.ts`: forwarding failures become `createSetupErrorMessage`, which creates a fresh empty assistant message with zero usage and no original diagnostics.
- `packages/coding-agent/src/core/agent-session.ts`: the prefix disables both ordinary retry and hard-error model fallback.
- `packages/coding-agent/src/core/credential-pool/classify.ts`: network-text matching does not include this WebSocket error.
- `packages/ai/src/api/openai-codex-responses.ts`: immediate SSE fallback is allowed only before WebSocket stream start. Subsequent requests can use the 60-second fallback cooldown; that does not recover the interrupted response.

## Expected behavior / requested investigation

1. Preserve the original partial assistant message, usage, and transport diagnostics across credential rotation and lazy-stream forwarding failures.
2. Distinguish harmless block-start bookkeeping from committed text or tool output when deciding whether recovery is safe. Keep the existing replay guard for actual committed output.
3. Classify abnormal WebSocket closure explicitly and define bounded recovery for safe pre-output cases.
4. Show an actionable interruption message without leaking the internal `senpi:no-turn-retry:` marker into ordinary user-facing error text.

I am not requesting unconditional replay after partial tool output: that would defeat the existing duplicate-side-effect protection.

## Related upstream work

- https://github.com/code-yeongyu/senpi/pull/1155 introduced the credential rotation/replay guard.
- https://github.com/code-yeongyu/senpi/pull/600 restores WebSocket probing after the SSE cooldown, while preserving the post-start replay guard.
- https://github.com/code-yeongyu/senpi/pull/330 handles a different Codex upstream error path.
- https://github.com/code-yeongyu/senpi/issues/683 concerns silent Codex hangs after tool results, not this explicit terminal error.

Targeted searches of open and closed issues/PRs in Senpi and OMO found no exact duplicate. The v2026.9.11-to-main comparison checked on 2026-09-12 contained no changes to the Codex WebSocket adapter, credential-pool implementation, or lazy-stream module.

## Unknowns

1006 alone cannot identify whether the remote provider, a network intermediary, local connectivity, or client runtime caused the abnormal closure. No packet capture, server logs, or event-loop timing trace was collected. A separate earlier log entry explicitly mentions a CloudFlare WebSocket proxy restart, but it does not establish that the 1006 incidents have the same cause.

## Self-contained reproduction script

```js
import assert from "node:assert/strict";
import { pathToFileURL } from "node:url";
import { resolve } from "node:path";

const root = resolve(process.argv[2]);
const ai = `${root}/node_modules/@earendil-works/pi-ai/dist`;
const { lazyStream } = await import(pathToFileURL(`${ai}/api/lazy.js`));
const { streamWithCredentialRotation } = await import(
pathToFileURL(`${root}/dist/core/credential-pool/rotation-stream.js`)
);
const { classifyCredentialFailure } = await import(
pathToFileURL(`${root}/dist/core/credential-pool/classify.js`)
);

const model = {
id: "diagnostic-model",
provider: "openai-codex",
api: "openai-codex-responses",
};
const sources = {
providerId: model.provider,
policy: { affinity: false, slots: { fixture: { env: "FIXTURE_KEY" } } },
env: (key) => key === "FIXTURE_KEY" ? "not-a-real-credential" : undefined,
repository: {
listSlots: async () => ({}),
envCredentialRevision: async () => "fixture",
mutateSlotState: async () => {},
},
};
const errorText = "WebSocket closed 1006 Connection ended";
assert.equal(classifyCredentialFailure(new Error(errorText)).kind, "fail_request");
assert.equal(classifyCredentialFailure(new Error("ECONNRESET")).kind, "retry_same");

for (const stage of ["start-only", "thinking-start", "partial-text"]) {
let attempts = 0;
const providerError = {
role: "assistant",
content: stage === "partial-text" ? [{ type: "text", text: "already visible" }] : [],
api: model.api,
provider: model.provider,
model: model.id,
usage: {
input: 12, output: 3, cacheRead: 0, cacheWrite: 0, totalTokens: 15,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
},
stopReason: "error",
errorMessage: errorText,
timestamp: 0,
diagnostics: [{
type: "provider_transport_failure",
timestamp: 0,
error: { name: "Error", message: errorText },
}],
};
const stream = lazyStream(model, async () => streamWithCredentialRotation({
sources,
runAttempt: async () => {
attempts++;
return (async function* () {
yield { type: "start", partial: providerError };
if (stage === "thinking-start") {
yield { type: "thinking_start", contentIndex: 0, partial: providerError };
}
if (stage === "partial-text") {
yield {
type: "text_delta", contentIndex: 0,
delta: "already visible", partial: providerError,
};
}
yield { type: "error", reason: "error", error: providerError };
})();
},
}));
const events = [];
for await (const event of stream) events.push(event.type);
const result = await stream.result();
const marked = result.errorMessage.startsWith("senpi:no-turn-retry:");
assert.equal(marked, stage !== "start-only");
assert.equal(attempts, 1);
assert.deepEqual(result.content, []);
assert.equal(result.usage.totalTokens, 0);
assert.equal(result.diagnostics, undefined);
console.log(JSON.stringify({
stage, attempts, events, marked,
finalContent: result.content,
finalTokens: result.usage.totalTokens,
diagnosticsPreserved: result.diagnostics !== undefined,
}));
}
console.log("CONFIRMED: current suppression and error-information loss reproduced.");
```

Contributor guide

Open the contributing guide

Research direction

Run the supplied reproduce.mjs script against the installed package first, then trace lazyStream in packages/ai/src/api/lazy.ts through rotation-stream.ts, failover.ts, classify.ts, and agent-session.ts. Compare the event handling and error classification with the requested recovery rules. Done means partial output, usage, and diagnostics survive failures, block-start events do not suppress safe recovery, the existing committed-output guard remains, and the internal marker is not user-facing.

Written by the indexing model from the issue text.

Assessment

Tech stack
bun, typescript
Domain
api, backend-api-design, cli
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.