stacklok / stacklok/mecatl

OpenRouter multi-upstream routing poisons sessions: replayed reasoning encrypted_content rejected with invalid_encrypted_content (Azure) — add a request-time heal

Open
#344 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

What happened

Live session on OpenRouter (openai/gpt-5.6-sol, Responses API) died mid-conversation with a permanent 400:

POST "https://openrouter.ai/api/v1/responses": 400 Bad Request
{"message":"Provider returned error","code":400,"metadata":{"raw":"{
  \"error\": {
    \"message\": \"The encrypted content gAAA...BzU= could not be verified. Reason: Encrypted content could not be decrypted or parsed.\",
    \"type\": \"invalid_request_error\",
    \"param\": null,
    \"code\": \"invalid_encrypted_content\"
  }}","provider_name":"Azure","is_byok":false,"provider_error_code":"invalid_encrypted_content"}}

The first failure fired on a live turn (not a resume). Leaving the session and coming back with "continue" reproduces the same 400 on every attempt — the session is permanently bricked.

Root cause

Not a mecatl replay bug — the adapter side is correct per OpenAI's store:false manual-replay contract:

  • internal/adapter/openai/stream.go (translate, the reasoning output-item arm) captures the genuine opaque encrypted_content REPLAY blob (requested via Include: reasoning.encrypted_content).
  • internal/adapter/openai/request.go (assistantItems) replays it verbatim on the reasoning input item every turn.

The broken assumption is upstream: the party that decrypts the blob must be the party that minted it. Encrypted reasoning content is keyed per org/deployment. OpenRouter (is_byok:false, its own key pool) load-balances a model slug across upstreams (OpenAI direct + one or more Azure OpenAI deployments). A blob minted on turn N by upstream A cannot be decrypted by upstream B on turn N+1 → invalid_encrypted_content.

Because mecatl is stateless (full replay each turn, by design) and Message.Reasoning is persisted in the session snapshot, the poisoned blob is baked into the conversation: every retry replays it, llmresilience correctly classifies the 400 as permanent (surfaced verbatim, never retried), the run ends failed, and Session.Recover re-enters only to re-fail identically. Same family as the Azure Duplicate item found with id fc_N bug (fixed via ToolCall.ItemID) and the empty-text wire poison (PR #291) — "strict/heterogeneous upstream via OpenRouter breaks a replay invariant, session bricks."

Proposed fix: reactive request-time heal (the PR #291 precedent)

On a 400 whose provider error code is invalid_encrypted_content, retry the turn once with the reasoning items omitted from the replay (an empty Message.Reasoning already yields no reasoning item, so the omission is wire-legal). Cost: reasoning continuity is lost for that conversation; benefit: the session survives instead of bricking. Notes:

  • This is reactive, not proactive — there is no way to detect key mismatch before the upstream rejects.
  • The heal belongs in the adapter/resilience layer, NOT port.LLMRequest (provider-private semantics).
  • Consider persisting the "reasoning replay disabled" outcome for the session (or dropping the stale blobs from history on the healed turn) so every subsequent turn doesn't re-pay a failed attempt.
  • The heal must be narrowly keyed on the provider error code, not any 400 — a generic 400 heal would mask real request bugs.

Related

  • Prevention half (pin one upstream so blobs stay decryptable): see the OpenRouter provider-pinning issue filed alongside this one.
  • UX half (a permanent-4xx failed session should say "retrying won't help"): see the permanent-4xx dead-end issue filed alongside this one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read internal/adapter/openai/stream.go, especially translate's reasoning output-item handling, and internal/adapter/openai/request.go around assistantItems; then trace the adapter/resilience request path for provider errors. Implement a single narrowly keyed retry for invalid_encrypted_content that omits reasoning replay, while preserving normal 400 handling and avoiding repeated failed attempts on later turns.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.