agent-substrate / agent-substrate/substrate

Envoy optimization to enable Rust dynamic modules

Abierto
#1,330 3 comentarios 0 reacciones 0 asignados Ver en GitHub
area/network kind/feature
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

## Problem

On the Envoy ingress path, every request costs one `ResumeActor` RPC to ate-apiserver — including requests to an actor that is already `RUNNING` on a known worker. `singleflight` collapses only concurrent callers, never sequential ones, so a warm actor receiving `n` requests/sec generates roughly `n` RPCs/sec that all return the same answer.

The code names the cost already, at `cmd/atenet/internal/router/ingress/resumer.go:182`:

> the accepted cost of one control-plane RPC per hot actor

Agent workloads are exactly the shape that makes this expensive: a working set of warm actors, each receiving many requests. It also puts ate-apiserver on the critical path of every request, against a north-star of 100ms p95 activation
latency (`docs/architecture.md:111-113`).

We use ext_proc extensively to implement Substrate logic. Because ext_proc lives in a separate process, this imposes a cost per-request. For example, on the Ingress path, we have the following flow: `client → envoy → ext_proc → ate_api_server → DB backend`. so there are two goals, one is to reduce the hop like Envoy can directly call the `ate_api_server`, and the second goal is to see if a cache is available or not.

## Proposal

`Option 1`. Add an Envoy **Rust dynamic module** (`envoy.filters.http.dynamic_modules`, already supported by the pinned `envoyproxy/envoy:v1.39-latest`) that sits **in front of** the ext_proc filter as a cache, and does not replace it:

- **Cache hit** — publish the same `envoy.filters.listener.original_dst` dynamic metadata ext_proc would have published, mark the request, and `clear_route_cache()` so route selection lands on a route carrying `ExtProcPerRoute.disabled`. ext_proc is skipped.
- **Cache miss** — do nothing. ext_proc runs exactly as today, with its resumer, parking, singleflight, error contract and metrics intact. The module then learns the binding from the metadata ext_proc published.

The dataplane therefore never talks to `ate-apiserver`, holds no client credential, and cannot route anywhere ext_proc has not already routed. Removing the filter restores current behaviour exactly.

Shape of the change: three additive edits to `xds.go` (one route, one filter, one flag defaulting off). With the flag off, the generated xDS is byte-identical to today's. `ingress/`, `extproc/`, `resumer.go` and `parking.go` are untouched.

`Option 2`. Directly implement a Rust filter that will connect to the `ate-apiserver` and the cache is also maintained in the filter itself.

## Measurements

Prototype at https://github.com/botengyao/substrate/tree/envoy-rust-dynamic-module (`demos/envoy-rust-dynamic-module/`). Envoy v1.39.1, 50 hot actors, concurrency 8, 20s after warmup, load generated inside the container network, reproduced twice
within 2%. The baseline runs the real `atenet router` binary configured by the real `xds.go` over ADS.

| Arm | RPS | p50 | p95 | ResumeActor calls |
|---|---:|---:|---:|---:|
| ext_proc → Go router (today) | 3,735 | 2.11 ms | 2.53 ms | 87,231 |
| rust module, cache off | 4,170 | 1.89 ms | 2.26 ms | 105,423 |
| rust module, cache on, ext_proc removed | 44,817 | 0.16 ms | 0.28 ms | 361 |
| **rust module + ext_proc together** | **42,766** | **0.17 ms** | **0.30 ms** | **262** |

The decomposition matters more than the headline: removing the gRPC hop is worth +12%; not making the call is worth 12×. **The cache is the win, not Rust.** Rust is the only place to put a cache that can also skip the hop.

Laptop-VM numbers against a stub control plane — the shape transfers, the absolute figures do not.

## Prerequisites and open questions

Correctness does not depend on the cache being fresh — `atunnel.authorize()` (`internal/atunnel/ingress.go:492`) fails closed, rejecting any request for an actor the worker does not currently host with a 421 and `X-Ate-Assignment-Stale`. But a usable cache needs:

1. **Evict on that 421, and re-run the slow path within the same request.** A cache hit skips `ResumeActor`, which is what wakes a suspended actor — evict-and-fail would turn the TTL into a user-visible error window on exactly the request that should have triggered a cold resume. This is the one place the design can regress behaviour.
2. **Evict on upstream reset**, since a vanished worker produces no 421 at all.
3. **`X-Ate-Assignment-Stale` is currently forgeable.** atunnel's `ReverseProxy` (`ingress.go:130-150`) has no `ModifyResponse` stripping it from actor responses, so an actor could force a `ResumeActor` per response. One-line fix, needed before anything trusts the header.

Open questions for discussion:

- Is a pure-Go TTL cache in `resumer.go` the better first step? It captures most of the absolute saving with none of the unsandboxed risk, but structurally cannot skip the ext_proc hop (the client always calls). Worth landing first regardless?
- The same missing-cache shape exists on egress, where `egress.go:168` calls `GetActor` per CONNECT with no cache and no singleflight — see the TODO at `egress.go:167` referencing #592.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.