Menci / Menci/Floway

Affinity carriers this instance cannot authenticate are forwarded upstream and dissolve the routing pin with no attributable record

Open
#456 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
33
Avg merge
45m
Merged PRs (30d)
10

Description

Provenance. This report was written by Claude Code. A human operator asked me to
investigate a suspected defect, and this is my own summary of the mechanism, the
reproduction, the impact, and a fix direction — not a hand-authored bug report. I have
deliberately filed an issue rather than a PR, because the fix involves a design choice I
do not think an outside agent should make unilaterally. Please run your own Claude Code
over the claims below to verify and amend them; I have tried to mark precisely where I am
confident and where I am not. Line references are against 7c1e79f9.

Problem

AffinityCodec.unwrap returns { kind: 'foreign', value } for two situations that are not
alike:

  1. the value carries no Floway framing — genuinely someone else's blob, and forwarding it
    verbatim is correct;
  2. the value carries Floway's framing but this instance cannot authenticate it — a carrier
    this deployment plausibly issued, under a key or a domain that no longer matches.

The fail-open itself is right and I am not asking for it to change: a carrier this instance
cannot open may belong to a Floway it is chained behind, and stripping the trailer would
corrupt that case. codec_test.ts pins this deliberately.

The problem is that case (2) has three silent consequences and leaves no trace anywhere in
Floway's own surfaces, so an operator debugging it has nothing to look at.

Evidence

unwrap computes the discriminating signal and discards it. splitOpaqueTrailer
succeeding is what separates (1) from (2):

// packages/gateway/src/data-plane/chat/shared/affinity/carrier.ts:136-137
const framed = splitOpaqueTrailer(value, IV_BYTES + 16);
if (framed === null) return { kind: 'foreign', value };   // (1) no framing
// …
// :159-161 — (2) framed like ours, failed to authenticate
} catch { return { kind: 'foreign', value }; }

Downstream, both collapse to the same verdict and the value is written into the
upstream-bound payload:

// packages/gateway/src/data-plane/chat/shared/affinity/selection.ts:50 and :60
if (decoded.kind === 'foreign') return { kind: 'preserve', value: decoded.value };
// packages/gateway/src/data-plane/chat/responses/affinity/ingress.ts:145
if (projection.kind === 'preserve') replacement[location.slot] = projection.value;

Three separate behaviours hang off the single kind === 'owned' gate, and all three fail
open together:

// ingress.ts:108-110 — only owned blobs pin routing
if (location.decoded.kind === 'owned') { latestOwnedTarget = location.decoded.affinity;
  if (required) requiredTargets.push(latestOwnedTarget); }
// ingress.ts:123 — only owned synthetic items are removed before dispatch
synthetic: blobs.some(blob => blob.decoded.kind === 'owned' && blob.decoded.syntheticItem === true),
// selection.ts:123-129 — the routing-unavailable guard is keyed on requiredTargets

With requiredTargets empty, the guard is dead and the consistency assertion at
selection.ts:81-84 is vacuously satisfied.

Nothing records the event: there is no log, metric or counter anywhere under
data-plane/chat/shared/affinity/ or data-plane/chat/responses/affinity/. Dumps do not
help either — respond.ts:56-57 runs observeResponsesFrames before
wrapResponsesClientEgress, so the dump holds pre-wrap response frames, and DumpRecord
(packages/gateway/src/dump/types.ts) has no upstream-request record at all. There is no
stored artifact of what Floway handed either party.

Reachability

This is the part I most want checked, because it is what decides whether this is a defect
or a diagnostic nicety.

The AES-GCM branch is entered whenever the key or the AAD differs from wrap time. The AAD
is domain ‖ originalBytes (carrier.ts:94-98, 126), and the domain is
responses.<item.type>.<slot> (ingress.ts:43-47). So it is reached by ordinary
operational events, not only by tampering:

  • a server-secret change — rotation, or a restore through the control-plane
    data-transfer path, or two instances that do not share the same secret;
  • a carrier-domain change — any change to item-type canonicalization, or to which
    items are treated as affinity carriers, invalidates every carrier already in a client's
    conversation history. Draft PR #273 changes exactly that set, which is part of why I am
    raising this now;
  • a client replaying an older conversation across either of the above.

Clients hold these blobs in conversation history and replay them for a long time, so the
window is not short.

What this issue does not claim

An earlier draft of this analysis argued that two of unwrap's exits are reachable only
after a successful authenticated decrypt and are therefore "provably this gateway's own".
On checking, that is wrong and I am withdrawing it:

  • carrier.ts:156 (origin undefined with non-empty original) is provably
    unreachable
    . wrap emits origin exactly when a value was supplied, and binds that
    value's bytes into the AAD, so origin === undefined implies the authenticated original
    was empty. Reaching it would require an AES-GCM forgery.
  • carrier.ts:152 (parseAffinityData returning null after a successful decrypt) is not
    reachable from production code; it needs a malformed AffinityTarget that the type
    system prevents. It becomes reachable only under version skewhasOnlyKeys
    (carrier.ts:28-29) is a closed check, so a carrier minted by a newer build carrying a
    new AffinityData key is rejected by an older one after a rollback. That is worth
    knowing, but it is a forward-compatibility property, not a current bug.

I also make no claim that this produces the invalid_encrypted_content rejections that
prompted the investigation. Which dimension chatgpt.com binds encrypted reasoning content
to is not publicly documented and I could not establish the link.

Impact

When case (2) occurs, all of the following happen together and none are visible:

  1. the affinity pin dissolves — requiredTargets is empty, so any viable candidate may
    serve a turn that was pinned to a specific upstream, and the routing-unavailable guard
    never fires;
  2. a synthetic carrier item — 100% Floway ciphertext, zero upstream-origin bytes — survives
    the removal at ingress.ts:123 and is transmitted to a third-party provider;
  3. an ordinary carrier reaches the upstream with Floway's trailer still attached.

The same selection.ts preserve path is shared by the chat-completions, messages and
gemini ingress modules, so this is not Responses-specific.

Fix direction

The behaviour to keep is the pass-through; the thing to recover is attribution. The
smallest change that does that is to carry the reason on DecodedAffinityBlob — still
kind: 'foreign', still byte-identical on the wire — and report it one level up, where
there is a request context to correlate against.

Three constraints I would ask any fix to respect, all of which caught me out:

  • The framing gate is not evidence of Floway authorship. splitOpaqueTrailer accepts
    when the trailing uint16be lands in [28, N-2], so for an arbitrary base64 blob the
    false-positive rate is (min(N-2, 65535) - 27) / 65536 — size-dependent, not constant.
    That is ~0.7% at 500 bytes, ~6% at 4 KB, ~15% at 10 KB, and ~100% above 64 KB. Reasoning
    encrypted_content sits in the KB range, so reporting per-blob on a framing hit would
    misattribute a large fraction of genuinely foreign values.
  • unwrap runs once per blob per item with no bound on item count, and the input is
    client-supplied. Any per-occurrence emission is a client-controlled amplifier; this wants
    aggregation — one record per request with a count — rather than one line per blob.
  • carrierDomain interpolates unvalidated client input. canonicalItemType
    (ingress.ts:43-44) only remaps compaction_summary and otherwise passes item.type
    through verbatim, so the domain string can contain arbitrary client-chosen characters
    including newlines. It must be escaped and length-bounded before it reaches any log or
    record.

A counter on the existing telemetry surface, keyed by reason, may serve better than a log
line — that is the design call I would rather leave to you.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/gateway/src/data-plane/chat/shared/affinity/carrier.ts and trace unwrap into selection.ts and responses/affinity/ingress.ts, then review codec_test.ts for the required pass-through behavior. Confirm how framed authentication failures are distinguished, aggregated per request, and safely reported without changing forwarded bytes; done means the existing fail-open behavior remains covered while the event becomes attributable.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.