Bug: A timed-out or refused upstream request is not flagged as an error on any abctl screen

Open
#1,045 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go

Research direction

Start with authbridge/authlib/listener/forwardproxy/server.go:491-495 and authlib/listener/reverseproxy/server.go:605-611, then trace the response-event handling in authbridge/authlib/usage/usage.go. Reproduce with the 300ms client timeout against the hanging upstream. Done means transport failures produce response-phase events carrying the 502 and an error in both listeners, so the event, errors, status, and latency views reflect the failure.

Written by the indexing model from the issue text.

Description

bug
Summary

When an outbound request fails at the transport level — a timed-out LLM call, connection refused, DNS failure, TLS failure — the client correctly receives a 502, but no response-phase session event is recorded. The failure is therefore invisible on every abctl screen: it cannot be distinguished from a request that is still in flight.

Reproduction

A hanging upstream behind a proxy whose client times out (probe against authlib/listener/forwardproxy, an empty pipeline, Client: &http.Client{Timeout: 300ms}, upstream sleeping 3s):

client saw status: 502
events recorded: 1
  phase=request  status=0  host="127.0.0.1:59946"  error=<nil>

One event, phase=request. No response event, so nothing carries the 502 and nothing carries an error.

Cause

authbridge/authlib/usage/usage.go — the request event is appended at the end of the request phase, but the response event (the one carrying StatusCode and Error) is appended much later. The upstream failure path returns between the two:

authbridge/authlib/listener/forwardproxy/server.go:491-495

resp, err := client.Do(r)
if err != nil {
	http.Error(w, `{"error":"bad gateway"}`, http.StatusBadGateway)
	return
}

The 502 exists only on the wire to the client, never in the session store. reverseproxy's errorHandler (authlib/listener/reverseproxy/server.go:605-611) has the same shape.

What each screen shows
Screen Behaviour
Events table One req row with STATUS blank (statusCell returns "" for StatusCode == 0), DURATION blank, no paired resp row — identical to a request still in flight
Usage → ERRORS Not counted. usage.go:563 sets Errors = 1 only for StatusCode >= 400 or SessionDenied; a timeout is neither
Usage → breakdown by status Absent. The label is added only when StatusCode > 0, so the request falls into the (unlabelled) remainder
Usage → latency Excluded — a zero duration is treated as "not measured" rather than as a failure
Detail pane The request JSON, with no error field and no response
Why it matters

The request event is counted in Requests, so a burst of timeouts renders as ordinary traffic with a healthy error rate. The usage pane is least informative exactly when an operator is trying to find out whether their upstream is failing — and "STATUS blank" reads as "still waiting", which is the wrong conclusion.

SessionEvent already has an Error field (pipeline.DeriveError), but it is only ever populated on the response path.

Suggested fix

Record a response-phase event with StatusCode: 502 and a populated Error before the early return, in both listeners. That would surface the failure in the events table, the ERRORS metric, and the status breakdown at once, without any change to those consumers.

An Error value distinguishing why the upstream call failed (timeout vs refused vs TLS) would be more useful than the status code alone, since all of them surface as 502 today.

Scope of this report

Verified for proxy-sidecar's forward proxy — the laptop / Claude Code path. reverseproxy looks the same by inspection but was not tested, and envoy-sidecar's ext_proc path was not examined at all, so I cannot say whether it records a response event on upstream failure.

Operating system and architecture

Darwin arm64

Cortex version

dev (traced at b838de5f; both cited files are unmodified from main)

Dominant language
Go
Stars
13
Forks
40
Avg merge
12h 17m
Merged PRs (30d)
156

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from rossoctl/cortex

All issues in rossoctl/cortex

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.