agent-substrate / agent-substrate/substrate

Envoy optimization to enable Rust dynamic modules

Ouverte
#1,330 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area/network kind/feature
Langage dominant
Go
Étoiles
1.8k
Forks
316
Merge moyen
2 j 43 min
PR mergées (30 j)
287

Description

## Problem

On the Envoy ingress path, every request costs one `ResumeActor` RPC to ate-apiserver — including requests to an actor that is already `RUNNING` on a known worker. `singleflight` collapses only concurrent callers, never sequential ones, so a warm actor receiving `n` requests/sec generates roughly `n` RPCs/sec that all return the same answer.

The code names the cost already, at `cmd/atenet/internal/router/ingress/resumer.go:182`:

> the accepted cost of one control-plane RPC per hot actor

Agent workloads are exactly the shape that makes this expensive: a working set of warm actors, each receiving many requests. It also puts ate-apiserver on the critical path of every request, against a north-star of 100ms p95 activation
latency (`docs/architecture.md:111-113`).

We use ext_proc extensively to implement Substrate logic. Because ext_proc lives in a separate process, this imposes a cost per-request. For example, on the Ingress path, we have the following flow: `client → envoy → ext_proc → ate_api_server → DB backend`. so there are two goals, one is to reduce the hop like Envoy can directly call the `ate_api_server`, and the second goal is to see if a cache is available or not.

## Proposal

`Option 1`. Add an Envoy **Rust dynamic module** (`envoy.filters.http.dynamic_modules`, already supported by the pinned `envoyproxy/envoy:v1.39-latest`) that sits **in front of** the ext_proc filter as a cache, and does not replace it:

- **Cache hit** — publish the same `envoy.filters.listener.original_dst` dynamic metadata ext_proc would have published, mark the request, and `clear_route_cache()` so route selection lands on a route carrying `ExtProcPerRoute.disabled`. ext_proc is skipped.
- **Cache miss** — do nothing. ext_proc runs exactly as today, with its resumer, parking, singleflight, error contract and metrics intact. The module then learns the binding from the metadata ext_proc published.

The dataplane therefore never talks to `ate-apiserver`, holds no client credential, and cannot route anywhere ext_proc has not already routed. Removing the filter restores current behaviour exactly.

Shape of the change: three additive edits to `xds.go` (one route, one filter, one flag defaulting off). With the flag off, the generated xDS is byte-identical to today's. `ingress/`, `extproc/`, `resumer.go` and `parking.go` are untouched.

`Option 2`. Directly implement a Rust filter that will connect to the `ate-apiserver` and the cache is also maintained in the filter itself.

## Measurements

Prototype at https://github.com/botengyao/substrate/tree/envoy-rust-dynamic-module (`demos/envoy-rust-dynamic-module/`). Envoy v1.39.1, 50 hot actors, concurrency 8, 20s after warmup, load generated inside the container network, reproduced twice
within 2%. The baseline runs the real `atenet router` binary configured by the real `xds.go` over ADS.

| Arm | RPS | p50 | p95 | ResumeActor calls |
|---|---:|---:|---:|---:|
| ext_proc → Go router (today) | 3,735 | 2.11 ms | 2.53 ms | 87,231 |
| rust module, cache off | 4,170 | 1.89 ms | 2.26 ms | 105,423 |
| rust module, cache on, ext_proc removed | 44,817 | 0.16 ms | 0.28 ms | 361 |
| **rust module + ext_proc together** | **42,766** | **0.17 ms** | **0.30 ms** | **262** |

The decomposition matters more than the headline: removing the gRPC hop is worth +12%; not making the call is worth 12×. **The cache is the win, not Rust.** Rust is the only place to put a cache that can also skip the hop.

Laptop-VM numbers against a stub control plane — the shape transfers, the absolute figures do not.

## Prerequisites and open questions

Correctness does not depend on the cache being fresh — `atunnel.authorize()` (`internal/atunnel/ingress.go:492`) fails closed, rejecting any request for an actor the worker does not currently host with a 421 and `X-Ate-Assignment-Stale`. But a usable cache needs:

1. **Evict on that 421, and re-run the slow path within the same request.** A cache hit skips `ResumeActor`, which is what wakes a suspended actor — evict-and-fail would turn the TTL into a user-visible error window on exactly the request that should have triggered a cold resume. This is the one place the design can regress behaviour.
2. **Evict on upstream reset**, since a vanished worker produces no 421 at all.
3. **`X-Ate-Assignment-Stale` is currently forgeable.** atunnel's `ReverseProxy` (`ingress.go:130-150`) has no `ModifyResponse` stripping it from actor responses, so an actor could force a `ResumeActor` per response. One-line fix, needed before anything trusts the header.

Open questions for discussion:

- Is a pure-Go TTL cache in `resumer.go` the better first step? It captures most of the absolute saving with none of the unsandboxed risk, but structurally cannot skip the ext_proc hop (the client always calls). Worth landing first regardless?
- The same missing-cache shape exists on egress, where `egress.go:168` calls `GetActor` per CONNECT with no cache and no singleflight — see the TODO at `egress.go:167` referencing #592.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.