ObolNetwork / ObolNetwork/obol-stack

Buyer is charged when an agent upstream fails but returns HTTP 200

Open
#821 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
11
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Summary

A buyer is charged when an agent upstream fails internally but still returns HTTP 200. Observed live on Base mainnet: an agent's LLM provider returned a non-retryable 404, the agent aborted its conversation loop, returned an empty reply with 200 application/json, and the payment settled on-chain. The buyer paid and received nothing.

Root cause

The settle decision is not "status < 400 after the response is known" — on the non-SSE path it happens before the body exists at all.

  • settlementInterceptor.WriteHeader calls settleFunc() at internal/x402/forwardauth.go:1311, i.e. the moment ReverseProxy hands over the upstream response headers. The only gate before it is statusCode >= 400 (internal/x402/forwardauth.go:1286).
  • The body-liveness guard that would catch an empty answer, bytesWritten == 0 (internal/x402/forwardauth.go:1347), is unreachable on this path: finalize returns immediately unless streamDefer is set (internal/x402/forwardauth.go:1324), and streamDefer is only set for text/event-stream (internal/x402/forwardauth.go:1297-1305).

So any upstream that reports its own failure inside a 200 is settled as a success.

Both request shapes are affected

  • stream: false — settles at WriteHeader, before any body byte. This is the shape that produced the incident.
  • stream: true (SSE) — reaches finalize, but a role-only delta plus data: [DONE] is roughly 60 bytes, which clears bytesWritten == 0 and settles too.

A fix that covers only one shape is not a fix.

It is recorded as a clean sale

chargedRequests increments (internal/x402/verifier.go:495-498) while upstreamFailedAfterVerify does not (internal/x402/verifier.go:499-503 requires status >= 400), so the failure is invisible in metrics.

Related defect found while investigating

settlementInterceptor.Flush (internal/x402/forwardauth.go:1362-1366) is an unconditional passthrough, and statusRecorder.Flush (internal/x402/verifier.go:1020-1024) forwards it to the real writer. httputil sets flushInterval = -1 for any response with ContentLength == -1 — chunked, not only SSE — and copyResponse arms delayedFlush before the first body byte.

A chunked application/json reply therefore commits an implicit 200 while wroteStatus is still false, which:

  • silently drops the X-PAYMENT-RESPONSE / PAYMENT-RESPONSE receipt set later at internal/x402/forwardauth.go:862-865, leaving the buyer to log "auth consumed without observed settlement" (internal/x402/buyer/proxy.go:741-748) — charged on-chain with no receipt to reconcile against;
  • defeats the wroteStatus guards at internal/x402/forwardauth.go:826 and :847, so a settle failure cannot flip the status to 503.

No existing test catches this: httptest handlers with small bodies get an automatic Content-Length, so flushInterval is 0. Any buffering scheme added here must gate Flush.

Proposed direction

Settle only when the upstream actually delivered a completion, behind an opt-in ForwardAuthConfig flag so existing behaviour is unchanged by default:

  1. For non-SSE chat-shaped routes, hold status and body until finalize and decide there.
  2. For SSE, tap the stream to detect whether any content delta was delivered — without buffering or delaying a byte, so incremental streaming timing is preserved.
  3. Treat a zero-byte body as "no value" before attempting to parse it — json.Unmarshal([]byte(""), …) returns an error, so a naive parse-then-fail-open settles the exact empty-body case in this report.
  4. Gate Flush under any buffering mode (see above).

Two-phase auth-capture (authorize now, capture once the answer is known good) is not available as an alternative: the facilitator rejects it with ERR_TWO_PHASE_NOT_SUPPORTED, and autoCapture: true is hardcoded at internal/x402/authcapture.go:142.

Impact

Any paid agent or inference offer whose upstream can fail while returning 2xx. With the platform fee enabled, the fee splits on-chain for the failed request too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with settlementInterceptor.WriteHeader, finalize, and Flush in internal/x402/forwardauth.go, then trace statusRecorder.Flush and the chargedRequests metrics in internal/x402/verifier.go. Reproduce both non-SSE and SSE paths, including chunked application/json responses, and add coverage showing settlement and receipt handling occur only after a valid completion while preserving streaming behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, payments
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.