TLS bridge auto-skip turns a CA trust error into 10-minute intermittent failures

Summary

When a client rejects the bridge's forged leaf, the TLS bridge auto-skips that host for 10 minutes and kills the current connection. The result is that a client which doesn't trust the bridge CA sees one hard failure per host per 10 minutes rather than a consistent failure — and every retry in between silently loses interception.

This is working as designed (self-healing for pinned clients), but the UX consequences are severe enough to be worth revisiting: it presents as network flakiness, it silently costs the parsing and tool-prune savings the bridge exists for, and it actively masks whether a CA fix worked.

Mechanism

authbridge/authlib/listener/forwardproxy/server.go:608-618:

tconn, err := s.TLSBridge.Term.Terminate(client, hostOnly(authority))
if err != nil {
    s.TLSBridge.Skip.Add(host) // pinned client → its retry will passthrough
    slog.Warn("tls-bridge passthrough", "host", host, "reason", "handshake-fail", "error", err)
    if s.bridgedRequests.Load() == 0 {
        s.noteBridgeHandshakeFailure()
    }
    return true // conn is dead post-forge; nothing left to tunnel
}

with skipTTL = 10 * time.Minute (authbridge/authlib/tlsbridge/decision.go:76).

So the cycle is:

1. First request to a host → bridge mints a leaf → client rejects it → hard x509 failure, host added to the skip set
2. Next 10 minutes → Classify returns Passthrough → every request succeeds, unparsed
3. TTL expires → next request fails again

Reproduction

Deterministic, 4/4 on a fresh host. probe is a Go client using the cortex CA as its only root, so success means the peer cert was cortex-signed (bridged) and failure means the real cert was presented (tunneled):

probe(pypi.org): BRIDGED (cortex-signed leaf)     ← fresh host, bridge intercepts
curl #1:         HTTP 000                         ← rejects leaf → hard fail + skip added
curl #2:         HTTP 200                         ← same command, now tunneled
probe(pypi.org): TUNNELED (real cert)             ← confirmed skipped

Confirmed on pypi.org, files.pythonhosted.org, proxy.golang.org, and registry.npmjs.org (the last stays BRIDGED when the client does trust the CA, which is the control).

Evidence this is happening in normal use

41 handshake-fail entries in one ~/.cortex/proxy.log, over several days:

  23 host=api.github.com
   3 host=go.opentelemetry.io
   3 host=github.com
   2 host=google.golang.org
   2 host=cafe.github.com
   1 host=raw.githubusercontent.com
   1 host=ghcr.io
   1 host=go.googlesource.com
   ...

api.github.com recurring 23 times is the signature of the 10-minute cycle against a Go tool (gh), which on macOS cannot be pointed at a CA file at all (see #911) and so rejects the leaf every time the TTL lapses.

Why this is worth changing

1. It presents as flakiness. One failure per host per 10 minutes reads as a transient network problem, not a misconfiguration. Nothing in the failing client's error (x509: certificate signed by unknown authority) points at the bridge or at the skip.

2. It silently costs what the bridge is for. Once skipped, the host tunnels — parsers and tool-prune stop seeing it, so both the session events and the token savings disappear with no error. There is a WARN for the "nothing was ever decrypted" case (noteBridgeHandshakeFailure), but nothing for per-host degradation on a bridge that is otherwise working.

3. It masks whether a fix worked. Because the retry succeeds, a broken configuration looks fixed. Concretely, while working on #911 this produced three false attributions: SSL_CERT_FILE appeared to fix gh, and GIT_SSL_CAINFO appeared to fix git, when in both cases the second attempt succeeded only because the first had just poisoned the skip set.

   The corollary matters for anyone testing CA config: "did the request succeed?" is not a valid success criterion. The valid one is "does the host stay BRIDGED after the client's request?"

4. The design conflates two different situations. A genuinely pinned client (cert-pinning app that will never trust any CA) should be skipped permanently. A client that simply is not configured yet is fixable, and skipping hides the thing the operator needs to know. Both take the same path today.

Possible directions

Not proposing a specific fix, but the options seem to be:

• Per-skip visibility. The WARN exists; nothing aggregates it. A count or a list of currently-skipped hosts on /stats, plus surfacing it in abctl, would make the degradation observable instead of inferable from a log grep.
• Distinguish pinned from unconfigured. Hard to detect from the server side, but consecutive-failure counting would separate "this host has failed 20 times over two days" (pinned, or a client that will never be fixed) from "first failure" (probably fixable).
• Reconsider re-attempting. A fixed 10-minute retry converts a one-time configuration error into indefinite periodic breakage. Backing off exponentially, or not re-attempting after N consecutive failures until a restart, would fail once loudly rather than forever quietly.
• Don't kill the connection. return true // conn is dead post-forge means the client's in-flight request is lost. If the bridge could decide to passthrough before forging (it cannot today — the rejection is the only signal), the first request wouldn't have to fail at all.

Environment

• macOS (darwin/arm64), cortex laptop service, tls_bridge.mode: enabled, generate_ca: true
• Reproduced against the running authbridge-proxy on 127.0.0.1:47600

Related

• #911 reduces how often this fires by making more clients trust the CA, but does not address the masking behaviour. Notably it cannot fix it for Go tools on macOS, where no environment variable can convey trust — so api.github.com will keep cycling until the CA is in the keychain.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com