agent-substrate / agent-substrate/substrate

[Bug]: atenet-router's 10s default route timeout cuts off long actor turns, and the 504 cannot be safely retried

Open
#1,525 0 comments 0 reactions 0 assignees View on GitHub
area/network kind/bug kind/docs prio/P0
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

### What happened?

`atenet-router` ships with `--route-timeout` unset, so Envoy's route to the actor falls back to
`defaultRouteTimeout = 10s`, with a further 5s of request parking in front of it. Any actor turn
that legitimately runs longer than that is cut off mid-flight and the caller gets
`504 upstream request timeout`.

This is what a fresh install does out of the box, with no unusual configuration. For an LLM agent
harness that relays a model completion, a turn running past 10s is the normal case rather than the
exception, so the effective ceiling on a stock cluster is 15s of work.

**The 504 cannot be safely retried.** It does not say whether the actor received the turn, and it
did: the actor is `RUNNING` and still working. Retrying runs the whole turn a second time. A correct
client therefore has to stop and report a failure, and the work still in progress is discarded.
From outside, it reads as the agent hanging, and nothing in the response points at a timeout an
operator could raise.

**The worse half is where the abandoned answer goes.** The actor finishes the work the 504 gave up
on, the response has nowhere to return to, and it is delivered to whoever speaks to that actor
next, as a fluent `HTTP 200` answering somebody else's question. In our run, 6 of 10 actors were
found holding a stale answer afterwards, and one of them surfaced internal harness framing into
user-visible text. Neither end observes an error, so a consumer cannot detect this without a turn
identifier the transport does not currently carry.

Observed as **10 failures out of 10** on a burst-wake test, each at 15.9–18.4s, which is the 5s
parking budget plus the 10s route timeout plus harness overhead.

### Expected Behavior

Two things, and the first is the release-relevant one.

1. **A stock install should not cap an actor turn at 10 seconds.** The default should be a value
that suits the workloads the platform is built for, or the install should refuse to come up
without an explicit choice.
2. **A response whose caller has gone away must never be delivered to a different caller.** A timed-out
turn should either be cancelled with the actor, or its response discarded. Handing it to the next
requester is a cross-request data leak, not just a lost turn.

Item 1 is a manifest line and a constant. The value to pick is already written down in the shipped
manifest, commented out, with the drain settings that move with it named in the same comment. See
Additional Context. Item 2 is the one that needs a design decision, and it should not hold up item 1.

### Steps to Reproduce

```
1. Install Substrate from the stock manifests, unmodified:
hack/install-ate.sh
(manifests/ate-install/atenet-router.yaml ships --route-timeout commented out)

2. Create an ActorTemplate for any workload whose request handler sleeps
longer than 15 seconds before responding. A 30s sleep is sufficient:

# in the actor's handler
time.sleep(30)
return "done"

3. Send one request to the actor through the atenet ingress and time it:

time curl -sS -o /dev/null -w '%{http_code}\n' \
http:/// -d '{"...": "..."}'

=> 504 upstream request timeout, at ~15s.

4. Confirm the actor is still working rather than dead:

kubectl-ate get actor -a agents actor-1 -o json | jq .status.state
=> ACTOR_STATE_RUNNING

5. Wait for the handler to finish, then send a SECOND, different request
to the same actor:

curl -sS http:/// -d '{"prompt": "say the word banana"}'

=> HTTP 200, and the body is the response to the request from step 3,
not to the one just sent.
```

### Sandbox Runtime

gVisor (runsc)

### Agent Substrate Version / Commit SHA

`c48b3a3c` (`release-0.1`). Also present on `main` at `941297cf`

### Kubernetes Version & Environment

GKE 1.35 (`v1.35.7-gke.1150000`), 5 nodes, us-central1-c

### Host OS & Architecture

Linux 6.6 (x86_64)

### Relevant Logs and Diagnostic Output

```shell
# Caller side, 10 concurrent turns to 10 distinct actors, stock install
status=504 elapsed=17.42s actor=actor-1 upstream request timeout
status=504 elapsed=16.88s actor=actor-2 upstream request timeout
status=504 elapsed=18.40s actor=actor-3 upstream request timeout
status=504 elapsed=15.91s actor=actor-4 upstream request timeout
... 10/10 identical, 15.9s - 18.4s

# The actors are alive and working the whole time
$ kubectl-ate get actors -a agents -o json | jq -r '.actors[].status.state'
ACTOR_STATE_RUNNING x10

# Next request to actor-1, several minutes later, asking an unrelated question
$ curl -sS "$INGRESS" -d '{"prompt":"say the word banana"}'
HTTP/1.1 200 OK
{"reply": ""}

# Source
cmd/atenet/internal/router/xds.go:147
const defaultRouteTimeout = 10 * time.Second
cmd/atenet/internal/router/ingress/parking.go:29
DefaultParkedRequestBudget = 5 * time.Second
```

### Additional Context

**The manifest already carries the fix, commented out, with a comment describing this exact
failure.** `manifests/ate-install/atenet-router.yaml:176-183`:

```yaml
# Envoy's end-to-end timeout on the workload route. Raise it for actors
# whose turns legitimately run long - a harness relaying an LLM
# completion holds the request open for the whole generation, and at the
# 10s default the client gets a 504 mid-turn. NOTE: the drain sequence
# deliberately does NOT scale with this; if you raise it and want long
# turns to survive a shutdown, raise --drain-timeout and
# terminationGracePeriodSeconds alongside it.
# - "--route-timeout=5m"
```

A default that is known to be wrong for the platform's target workload, shipped commented out, is
not really a default. **A comment three lines above the fix is not sufficient documentation, and we
are the evidence for that:** we read this file during install, left the default in place, and then
took 10 failures out of 10 on the first test that ran a long turn. If the team that wrote the
integration missed it, an operator installing from the quickstart will too.

Suggested fix, in order of preference:

1. Ship `--route-timeout=5m` uncommented in the stock manifest, and move `--drain-timeout` and
`terminationGracePeriodSeconds` with it as the comment already instructs.
2. Failing that, raise `defaultRouteTimeout` itself, since the constant is what a from-source or
Helm-templated install will pick up regardless of this manifest.

The response-to-the-wrong-caller behaviour is arguably a separate issue and can be split out if you
prefer; it is filed together because the timeout is what makes it reachable, and fixing only the
timeout leaves the leak in place for anyone whose turns exceed whatever the new value is.

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [ ] I have verified that this issue occurs on the latest commit on `main`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.