Bug: A timed-out or refused upstream request is not flagged as an error on any abctl screen
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 74/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- api, backend, observability
Research direction
Start with authbridge/authlib/listener/forwardproxy/server.go:491-495 and authlib/listener/reverseproxy/server.go:605-611, then trace the response-event handling in authbridge/authlib/usage/usage.go. Reproduce with the 300ms client timeout against the hanging upstream. Done means transport failures produce response-phase events carrying the 502 and an error in both listeners, so the event, errors, status, and latency views reflect the failure.
Written by the indexing model from the issue text.
Description
Summary
When an outbound request fails at the transport level — a timed-out LLM call, connection refused, DNS failure, TLS failure — the client correctly receives a 502, but no response-phase session event is recorded. The failure is therefore invisible on every abctl screen: it cannot be distinguished from a request that is still in flight.
Reproduction
A hanging upstream behind a proxy whose client times out (probe against authlib/listener/forwardproxy, an empty pipeline, Client: &http.Client{Timeout: 300ms}, upstream sleeping 3s):
client saw status: 502
events recorded: 1
phase=request status=0 host="127.0.0.1:59946" error=<nil>
One event, phase=request. No response event, so nothing carries the 502 and nothing carries an error.
Cause
authbridge/authlib/usage/usage.go — the request event is appended at the end of the request phase, but the response event (the one carrying StatusCode and Error) is appended much later. The upstream failure path returns between the two:
authbridge/authlib/listener/forwardproxy/server.go:491-495
resp, err := client.Do(r)
if err != nil {
http.Error(w, `{"error":"bad gateway"}`, http.StatusBadGateway)
return
}
The 502 exists only on the wire to the client, never in the session store. reverseproxy's errorHandler (authlib/listener/reverseproxy/server.go:605-611) has the same shape.
What each screen shows
| Screen | Behaviour |
|---|---|
| Events table | One req row with STATUS blank (statusCell returns "" for StatusCode == 0), DURATION blank, no paired resp row — identical to a request still in flight |
| Usage → ERRORS | Not counted. usage.go:563 sets Errors = 1 only for StatusCode >= 400 or SessionDenied; a timeout is neither |
| Usage → breakdown by status | Absent. The label is added only when StatusCode > 0, so the request falls into the (unlabelled) remainder |
| Usage → latency | Excluded — a zero duration is treated as "not measured" rather than as a failure |
| Detail pane | The request JSON, with no error field and no response |
Why it matters
The request event is counted in Requests, so a burst of timeouts renders as ordinary traffic with a healthy error rate. The usage pane is least informative exactly when an operator is trying to find out whether their upstream is failing — and "STATUS blank" reads as "still waiting", which is the wrong conclusion.
SessionEvent already has an Error field (pipeline.DeriveError), but it is only ever populated on the response path.
Suggested fix
Record a response-phase event with StatusCode: 502 and a populated Error before the early return, in both listeners. That would surface the failure in the events table, the ERRORS metric, and the status breakdown at once, without any change to those consumers.
An Error value distinguishing why the upstream call failed (timeout vs refused vs TLS) would be more useful than the status code alone, since all of them surface as 502 today.
Scope of this report
Verified for proxy-sidecar's forward proxy — the laptop / Claude Code path. reverseproxy looks the same by inspection but was not tested, and envoy-sidecar's ext_proc path was not examined at all, so I cannot say whether it records a response event on upstream failure.
Operating system and architecture
Darwin arm64
Cortex version
dev (traced at b838de5f; both cited files are unmodified from main)
- Dominant language
- Go
- Stars
- 13
- Forks
- 40
- Avg merge
- 12h 17m
- Merged PRs (30d)
- 156
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from rossoctl/cortex
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
nice to have
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
stale
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
Similar issues
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
githubnext/gh-aw-cao#13143 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
blinklabs-io/bursa#904 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
yanet-platform/ipfw-go#129 ·
-
bug confmap/provider/googlesecretmanagerprovider needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
open-telemetry/opentelemetry-collector-contrib#51273 · 2 comments ·
-
bug: AI Gateway client filter lists "Unknown" twice when NULL and literal Unknown clients coexist Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100