AI Gateway 1.2.0: intermittent HTTP 504 `upstream request timeout` on LLM proxy routes following an idle period

Reported: 2026-09-13
Severity (our view): High for demo/production use — a single occurrence stalls a caller for ~60 s
Related: wso2/api-platform issue #1466 (policy-engine panic on client disconnect) — related but not the same symptom; see §7

---

1. Summary

On a WSO2 API Platform AI Gateway 1.2.0 VM deployment, a minority of requests to an LLM
proxy route stall for ~60 seconds and then return HTTP 504 with the plain-text body
upstream request timeout and header server: WSO2 API Platform.

The same request repeated immediately afterwards succeeds in ~1.7–2.0 s. Failures are
intermittent and not deterministically reproducible. They are strongly associated with a
preceding idle period: under sustained traffic the gateway is stable (30/30 clean, max 2.7 s),
whereas the first few calls after a gap of ~10 min or more fail at a material rate. The
association is probabilistic, not deterministic — a call after a 25-hour idle succeeded in 2.7 s,
and in one run the first call after a 27-minute idle succeeded while the second failed.
Failures do not correlate with payload size, token count, model, or tool count.

A control arm rules out the model provider: the identical prompt sent to the same model
(gpt-4.1) through a different, managed WSO2 gateway over the same period
produced 0 failures in 20 calls, max 2,949 ms.

2. Impact

The gateway fronts five LLM agents in a live demo rig. One stalled call:

• consumes ~60 s of a 120 s end-to-end budget, and
• in one observed case the caller measured 125.6 s for a single chat completion that
  ultimately returned 200, which exceeded the orchestrator's 75 s agent-loop budget and
  forced a fallback path.

Because the stall presents as an unresponsive request rather than a fast error, callers cannot
distinguish it from a hung backend until their own timeout fires.

3. Environment

Component	Version / detail
Product	WSO2 API Platform AI Gateway 1.2.0 (VM install path, wso2apip-ai-gateway-1.2.0.zip, upgraded from 1.1.0 on 2026-09-12 via the shipped scripts/setup.sh)
Containers	gateway-controller + gateway-runtime (Envoy), shipped docker-compose.yaml + a local docker-compose.override.yaml (only restart: unless-stopped and a 443:8443 publish)
Control plane	Bijira, connect.bijira.dev · gateway id <gateway-id> · environment Development · manifest with 41 policies
Host	Azure Standard_D2s_v5 (2 vCPU / 8 GiB), Ubuntu 24.04, region southafricanorth
Container runtime	Docker Engine 29.8.0, Compose v5.5.1
Listener	TLS on host 443 → container 8443; Let's Encrypt ECDSA certificate (CN=<gateway-host>)
Route shape	proxy → provider chaining; the proxy forwards to the provider's route on the gateway's own internal port 127.0.0.1:8080, and the provider calls https://api.openai.com/v1
Upstream model	OpenAI gpt-4.1 (reported build gpt-4.1-2025-04-14)
Auth	X-API-Key: <key> (this gateway rejects Authorization: Bearer and api-key)
Clients that reproduce it	Python 3.12 urllib and Ballerina 2201.13.5 ballerina/http — i.e. not client-specific

4. Observed behaviour

4.1 Exact failure response

call 1 (after ~27 min idle): 2.0s   HTTP 200
call 2:                      60.8s  HTTP 504
                                    server: WSO2 API Platform
                                    content-type: text/plain
                                    body: "upstream request timeout"
call 3:                      1.7s   HTTP 200

Note call 1 succeeded and call 2 failed, with no idle gap between them.

4.2 A/B against a control gateway

20 alternating call pairs, identical prompt and model, same wall-clock window
(so any OpenAI-side variance affects both arms equally):

Arm	n	p50	p90	max	failures
This gateway (<gateway-host>)	20	1,742 ms	60,836 ms	60,964 ms	2 × HTTP 504
Control: a different managed WSO2 gateway, same model	20	1,354 ms	1,885 ms	2,949 ms	0

4.3 Latency observed from the application

Slow calls carry the same request shape as fast ones — this is not a payload-size effect:

call	duration	input tokens	output tokens	tools	finish reason	result
orchestrator	125,596 ms	484	21	4	tool_calls	200
orchestrator (next call)	999 ms	543	16	4	stop	200
identity agent	47,682 ms	492	44	2	tool_calls	200
identity agent (later)	1,693 ms	490	50	2	tool_calls	200

21 output tokens returned after 125 s indicates the time is not spent generating —
it is spent before any upstream response begins.

4.4 Idle traffic is the strongest association

Condition	Calls	Failures
Sustained traffic, back-to-back with sub-second gaps	30	0 (max 2.7 s)
First calls after ~11 min idle	20 (alternating, §4.2)	2 — both in the first two calls
First calls after ~27 min idle	3	1 — the second call; the first succeeded

This is the clearest signal we have: the gateway is stable once it is actively serving, and
failures cluster in the first few requests after an idle gap. The association is probabilistic
rather than deterministic — a request after a 25-hour idle returned in 2.7 s, and as the third row
shows the failing request is not necessarily the first one.

This pattern is consistent with a connection pool (either the proxy → provider loopback hop, or
the provider → OpenAI hop) holding entries that go stale while idle, where a request that picks a
stale entry waits out the ~60 s upstream timeout. We have not been able to confirm this from
gateway-side data (§8).

4.5 What does not correlate

• Payload size / token count / tool count — see §4.3.
• Model or route — reproduced on more than one per-agent proxy, all on gpt-4.1.
• Client library — reproduced from Python urllib and from Ballerina ballerina/http.
• Upstream provider — the control arm in §4.2 exonerates OpenAI for the same window.

5. Reproduction

Any HTTP client will do; the failure is server-side. The script below alternates nothing and
simply repeats one minimal request until a 504 appears (typically within a few dozen calls).

BASE='https://<gateway-host>/<project>/<proxy-name>'   # LLM proxy route
KEY='<api key>'

for i in $(seq 1 30); do
  start=$(date +%s)
  code=$(curl -s -o /tmp/out.$$ -w '%{http_code}' --max-time 180 \
    -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
    -d '{"model":"gpt-4.1","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":5}' \
    "$BASE/chat/completions")
  printf 'call %2d  %3ss  HTTP %s  %s\n' "$i" "$(( $(date +%s) - start ))" "$code" "$(head -c 60 /tmp/out.$$)"
done

Expected: every call returns 200 in 1–3 s.
Actual: a minority of calls return HTTP 504 upstream request timeout after ~60 s.
Observed rate in our measurements: 2 / 20 and 1 / 3 — both on gateways that had been
idle beforehand (§4.4). To reproduce reliably, leave the gateway idle for ≥ 10 minutes first,
then issue several calls; back-to-back traffic on an already-active gateway did not reproduce it
(0 / 30).

6. What we would like to understand

The evidence in §8 narrows this considerably: the request is dispatched to the upstream
provider, and then zero response bytes arrive for 60 s, on a route that answers in ~1.8 s
before and after. A second gateway to the same OpenAI endpoint, over the same window, never
reproduced it (§4.2). Our questions:

1. Is the stall inside the gateway rather than at OpenAI? The policy engine emits no log line
   for the stalled request until the 60 s timeout fires (§8.3), whereas successful requests are
   logged as they pass. Could a request be held before or inside policy processing while Envoy has
   already opened the upstream connection?
2. Connection reuse after idle. Upstream idle_timeout is 300 s and every failure we have
   followed a longer gap. Is there a known race where a connection selected from the pool after
   idle is no longer usable, such that the request is written but never progresses?
3. Should the 60 s route timeout be tunable per route? A stalled request currently occupies a
   caller for a full minute; failing fast would let clients retry (a retry succeeds immediately
   in our measurements).
4. Is the #1466 panic harmful here, or purely cosmetic? It fires on every abandoned request
   (§8.3). If it leaves the policy engine in a degraded state it may explain why failures
   sometimes cluster.

7. Relationship to issue #1466

wso2/api-platform #1466 reports a policy-engine panic (Recovered from panic in Process method) triggered by client disconnect. We previously believed 1.2.0 resolved our hangs, then
corrected that: 1.2.0 removed a trigger, not the panic, and the panic still fires whenever a
request is abandoned.

This report is a distinct symptom and should probably be a separate issue:

	#1466	This report
Symptom	request never answered; connection held	explicit HTTP 504 after ~60 s
Envoy accounting	downstream_cx_destroy_remote_active_rq, no 5xx	5xx returned to the client
Trigger	client disconnects	occurs with the client still waiting

The two may share a root cause, but a caller sees different behaviour and needs different handling.

8. Gateway-side evidence (collected 2026-09-13)

Captured on the VM while reproducing one failure. Happy to attach the raw files on request.

8.1 Envoy access log — the two hops of one failing request

Correlation id 2155282e-6dcb-47ff-a863-e13a1e1875ec:

hop 1 (client -> proxy route)
[2026-09-13T10:10:35.428Z] "POST /<proxy-route>/chat/completions HTTP/1.1"
  -> /<provider-route>/chat/completions HTTP/1.1
  504 UT response_timeout - 106 24 60003 1 - 0 -   "127.0.0.1" "127.0.0.1:8080"

hop 2 (provider route -> OpenAI)
[2026-09-13T10:10:35.430Z] "POST /<provider-route>/chat/completions HTTP/1.1"
  -> /v1/chat/completions HTTP/1.1
  0 DC downstream_remote_disconnect - 106 0 60000 0 - 0 -   "api.openai.com" "172.66.0.243:443"

Reading:

• Hop 2 shows response code 0, flag DC, and upstream_service_time = - — the
  request was sent upstream and no response headers ever arrived, for the full 60 s.
• Hop 1 hit its 60 s route timeout (UT response_timeout) 3 ms later and returned 504,
  which tore down hop 2's connection (hence DC, not a timeout of its own).

A successful request moments later on the same route and the same upstream IP took 1.8 s.

8.2 Envoy counter delta across exactly one reproduced failure

counter	proxy -> provider (127.0.0.1:8080)	provider -> OpenAI
upstream_rq_timeout	3 -> 4 (+1)	unchanged
upstream_rq_5xx	3 -> 4 (+1)	unchanged
upstream_cx_destroy_local_with_active_rq	18 -> 19	18 -> 19
upstream_cx_total	23 -> 24	27 -> 29

The OpenAI cluster records no timeout and no 5xx — its connection was abandoned when the
inner hop timed out, not failed on its own terms.

8.3 Policy engine

On successful requests the policy engine logs immediately (two No valid trace context extracted lines as the request passes). For the stalled request it logged nothing until the
60 s mark, then:

[pol] 10:11:35.431 INFO  Template handle(extracted from route metadata): templateHandle=openai
[pol] 10:11:35.957 ERROR panic occurred
                         error="runtime error: invalid memory address or nil pointer dereference"
                         context="Recovered from panic in Process method"

This is the #1466 panic. Note it fires at the 60 s timeout, i.e. as a consequence of the
request being abandoned — not as the cause of the stall.

8.4 Relevant configuration (from /config_dump)

"timeout":         "60s"    (x74)   <- route timeout; matches the observed 60.0-60.8 s failures
"idle_timeout":    "300s"   (x72)   <- upstream connection idle timeout
"connect_timeout": "5s"

Every failure we recorded followed a gap longer than the 300 s upstream idle timeout; 30
back-to-back calls with sub-second gaps produced none.

8.5 Versions confirmed on the host

gateway-runtime    ghcr.io/wso2/api-platform/gateway-runtime:1.2.0
gateway-controller ghcr.io/wso2/api-platform/gateway-controller:1.2.0
Envoy              1.38.3 (0ebfcfe5b0484b89ca85b761da9e05ce75dbda8d/Clean/RELEASE/BoringSSL)
Ubuntu             24.04.4 LTS

9. Workarounds we are adopting

Pending a fix, the client side will:

1. cap each LLM call well below the gateway's 60 s (~15 s) and retry once — the evidence is
   that an immediate retry succeeds;
2. issue a sacrificial warm-up call before a user-visible operation, so any stall is absorbed
   off the critical path;
3. keep a rollback path to the managed gateway of §4.2 as a fallback.

Start by reproducing the idle-period failure with the curl loop, then compare the Envoy access logs and counter deltas for the proxy-to-provider and provider-to-OpenAI hops. Read the policy engine Process path and its timeout-related logs, using the captured 504 and panic as the baseline; done means identifying the failing component and providing a verified fix or documented mitigation.