agent-substrate / agent-substrate/substrate

[Bug]: A checkpoint cancels the actor's in-flight requests and returns a bare 502

未关闭
#1,572 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
area/node kind/bug prio/P1
主要语言
Go
星标
1.8k
派生
316
平均合并
2 天 43 分钟
30 天内合并 PR
287

描述

### What happened?

Suspending an actor that is in the middle of serving a request kills the request. The caller gets
`502` with a 12-byte body, `bad gateway\n`, and no indication that the actor is healthy, that a
checkpoint was in progress, or that a retry would work.

This is not a race against a narrow window. `CheckpointWorkload` calls `deactivateActorNetworking`
as its first step, before it touches the sandbox. That calls `atunnel.Server.Deactivate`, which
cancels the context of every in-flight request and then waits for the handlers to exit. The
cancelled request unwinds through the `httputil.ReverseProxy` `ErrorHandler` in
`internal/atunnel/ingress.go`, which does `http.Error(w, "bad gateway", http.StatusBadGateway)`.
The wait after the cancel is for cleanup, not a drain, so nothing gives the request a chance to
finish first.

So any request still open when a suspend reaches the worker is cancelled, deterministically, and
`502` is the only thing it can return.

We measured this against an agent workload whose turns take several seconds. Twelve trials, a
request issued at 0, 0.25, 0.5, 0.75, 1.0 and 1.5 seconds after the suspend call, two sweeps:

| Request issued at | Status | 502 arrives after | Absolute time of the 502 |
|---|---|---|---|
| 0.00 s | 502 | 1.41 s | 1.41 s |
| 0.25 s | 502 | 1.26 s | 1.51 s |
| 0.50 s | 502 | 0.97 s | 1.47 s |
| 0.75 s | 502 | 0.73 s | 1.48 s |
| 1.00 s | 502 | 0.56 s | 1.56 s |
| 1.50 s | 200 | n/a | served normally |

The last column is the point. The `502` lands at a fixed moment in wall-clock time, roughly 1.5 s
into a suspend call that takes about 1.9 s, regardless of when the request was sent. That is a
single cancellation event, not a window being crossed.

The 1.5 s trials succeed because by then the actor is `SUSPENDED`, so the router parks the request
and resumes the actor on the normal path. That path works. The defect lives entirely in the
interval where the control plane still reports `ACTOR_STATE_RUNNING` and atunnel has already been
deactivated, so the router forwards straight through to a worker that will not serve.

Actor memory is unaffected. The checkpoint itself is clean and the actor restores correctly in
every trial.

The same code is on `main` (`67304608`) and `release-0.1`.

### Expected Behavior

A suspend should not destroy work that is already in flight.

`Deactivate` should drain: let in-flight requests complete against a budget before cancelling them.
If the budget expires and a request must be cancelled, it should come back as a retryable status
with `Retry-After` rather than a bare `502`, so a client can do the right thing without having to
guess whether the actor is broken.

Either half is an improvement on its own. The drain is the one that keeps the request alive.

### Steps to Reproduce

Any actor with a handler that takes longer than the suspend call reproduces this. A one-line sleep
handler is enough.

```shell
# 1. An actor whose HTTP handler holds the request open for 10s.
kubectl ate create actor actor-1 -a agents --template-ref slow-handler
kubectl ate resume actor actor-1 -a agents

# 2. Issue a request that will still be open when the suspend lands, then
# suspend the actor while it is running.
curl -sS -w '\n%{http_code} after %{time_total}s\n' \
http://actor-1.agents.actors.resources.substrate.ate.dev/slow &
sleep 0.5
kubectl ate suspend actor actor-1 -a agents
wait

# 3. The curl returns 502 with a 12-byte body, at a fixed point in the suspend
# call rather than at a fixed delay after the request.
#
# 4. Confirm the actor is fine: it resumes and serves normally.
kubectl ate resume actor actor-1 -a agents
curl -sS http://actor-1.agents.actors.resources.substrate.ate.dev/slow
```

Sending the request *after* the actor reaches `SUSPENDED` is the working case, not a reproduction:
the router parks it and resumes the actor.

### Sandbox Runtime

gVisor (runsc)

### Agent Substrate Version / Commit SHA

`c48b3a3c` (`release-0.1`). The same code paths are present on `main` at `67304608`.

### Kubernetes Version & Environment

GKE 1.35 (`v1.35.7-gke.1150000`), us-central1-c

### Host OS & Architecture

Linux 6.6 (x86_64)

### Relevant Logs and Diagnostic Output

```shell
# Caller, request issued 0.5s before the suspend:
< HTTP/1.1 502 Bad Gateway
< content-type: text/plain; charset=utf-8
< content-length: 12
bad gateway

# Worker, at the moment of the 502. This is the ErrorHandler firing on the
# cancelled request context, not an upstream failure:
level=WARN msg="atunnel upstream request failed" err="context canceled"

# The body is Go's http.Error, which appends the newline, hence exactly 12 bytes.
# Note this collides with the string kubectl port-forward emits on stream
# failure, which is worth knowing when reading a report of this: we ruled that
# out separately before finding the source.

# Source, at c48b3a3c:
# cmd/ateom-gvisor/main.go:737 CheckpointWorkload
# cmd/ateom-gvisor/main.go:746 -> deactivateActorNetworking, first step
# internal/atunnel/ingress.go:435 Deactivate: cancel, then wait for handlers
# internal/atunnel/ingress.go:146 ReverseProxy ErrorHandler
# internal/atunnel/ingress.go:148 http.Error(w, "bad gateway", 502)

One thing that would help independently of the fix: a counter on the cancel path in `Deactivate`,
so operators can see how often a suspend is landing on live traffic. Today there is no signal at
all, and a soak that only sends to already-suspended actors will report zero occurrences of this
no matter how often it happens in real use.
```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [ ] I have verified that this issue occurs on the latest commit on `main`.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。