Eager Upstream Connections: per_upstream_min_connections and Connection-Aware Load Balancing

This proposes two independent, complementary cluster-level features that together bring Envoy's upstream connection behavior in line with gRPC's client-side connection management model:

1. **`PreconnectPolicy.per_upstream_min_connections`** — a per-host minimum on open connections. When non-zero, Envoy proactively establishes connections so the configured floor is maintained, without waiting for a request to arrive.
2. **`Cluster.connection_aware_load_balancing`** — when enabled, host selection prefers hosts that already have at least one ready connection on the current worker thread, falling back to the underlying load balancer's choice if none does.

The two are designed to be configured independently. The first eliminates the connection-establishment latency on the request path. The second closes the brief race window where a connection is being established when a request arrives, by routing preferentially to hosts that are already ready.

Disclosure: I used GenAI to help design and implement this proposal and the accompanying impl. I have manually reviewed it all myself

## Motivation

### The Problem

Envoy's current upstream connection model is fundamentally **lazy/on-demand**. The request lifecycle is:

1. Router filter calls `chooseHost()` on the load balancer
2. LB selects a host based on health/weight — **without considering connection state**
3. `httpConnPool()` obtains or creates a connection pool for the selected host
4. `newStreamImpl()` checks for ready connections; if none exist, it creates a new one
5. **The request blocks while the TCP + TLS handshake completes**

This means:
- The **first request to any new host** pays the full connection establishment cost
- When a **connection is lost** (GOAWAY, idle timeout, max lifetime), the LB continues selecting that host, and the next request blocks on reconnection
- During **cluster cold start**, every initial request to every host blocks on connection establishment

### Why the Existing `preconnect_policy` Is Insufficient

Envoy already has a preconnect feature ([`PreconnectPolicy`](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto)) with two settings:

- **`per_upstream_preconnect_ratio`**: Establishes extra connections to the *same* upstream host, anticipating follow-up streams. For example, a ratio of 1.5 with 100 active streams would maintain 50 preconnected connections.
- **`predictive_preconnect_ratio`**: Preconnects to *different* upstream hosts across the cluster, predicting which host the LB would pick next.

Both are **reactive** — they only fire during request processing inside `newStreamImpl()` and `ClusterManagerImpl::maybePreconnect()`. They cannot:

1. **Pre-warm connections to newly discovered endpoints**: When EDS delivers new hosts, no connections are established until the first request is routed to each host.
2. **Establish connections to idle clusters**: A cluster with no traffic has zero connections, regardless of preconnect configuration.
3. **Recover connections lost to GOAWAY or idle timeout**: When a connection closes, preconnect doesn't re-establish it until the next request arrives for that host.
4. **Prevent the first request from blocking**: Preconnect only helps after at least one connection exists; it cannot eliminate the initial connection establishment latency.

Additionally, preconnect is capped at 3 connection attempts per request arrival (`tryCreateNewConnections()` loops at most 3 times), and `predictive_preconnect_ratio` is limited to one preconnect per configured upstream.

The feature proposed here is **complementary** to `preconnect_policy`. Preconnect optimizes steady-state connection pool sizing; eager connection establishment ensures connections exist in the first place.

Prior requests for this capability — [#21346](https://github.com/envoyproxy/envoy/issues/21346) ("Pre-initialize upstream connection pools") and [#2755](https://github.com/envoyproxy/envoy/issues/2755) ("Pre-establish upstream connections").

### How gRPC Solves This

gRPC's client-side load balancing model (specifically `round_robin` and `weighted_round_robin`) takes a fundamentally different approach:

#### Subchannel State Machine

Each endpoint gets a **subchannel** with states: `IDLE → CONNECTING → READY → TRANSIENT_FAILURE → SHUTDOWN`

#### Proactive Connection Establishment

When the name resolver returns a new set of addresses, `round_robin` immediately initiates a connection to each one. The connection attempts happen asynchronously in the background — no RPC is required to trigger them. See the comment in [`RoundRobinLoadBalancer.java#L55`](https://github.com/grpc/grpc-java/blob/master/util/src/main/java/io/grpc/util/RoundRobinLoadBalancer.java#L55):

```java
// RRLB will request connection immediately on subchannel IDLE.
```

#### READY-Only Routing

The picker built by `round_robin` **only includes READY subchannels**. See [`RoundRobinLoadBalancer.java#L56`](https://github.com/grpc/grpc-java/blob/master/util/src/main/java/io/grpc/util/RoundRobinLoadBalancer.java#L56):

```java
List activeList = getReadyChildren();
// ...
updateBalancingState(READY, createReadyPicker(activeList));
```

RPCs to non-READY subchannels return `PickResult.withNoResult()` — the RPC is **buffered**, not failed.

#### Auto-Recovery

When a subchannel transitions to IDLE (connection lost), `round_robin`'s `ChildLbStateHelper` calls `getLb().requestConnection()` to reconnect in the background. The subchannel is removed from the picker during reconnection.

### The Result

In gRPC's model:
- **No RPC ever blocks on connection establishment** (they are queued until a subchannel is READY)
- **New endpoints get traffic only after connections are established**
- **Connection loss is recovered in the background** — traffic redistributes to remaining READY subchannels
- **Cold start is amortized** — all connections are established in parallel during channel initialization

### Real-World Impact

We have observed significant tail latency regressions when migrating services from gRPC client-side load balancing to Envoy-based service mesh proxying. The root cause is this architectural mismatch: gRPC clients maintain warm connections to all discovered endpoints and only route to ready subchannels, while Envoy establishes connections lazily on the request path. The latency impact is particularly acute for:

- **Latency-sensitive gRPC services** where even 5-15ms of connection setup on the request path is unacceptable
- **Low-RPS services** where connections idle out frequently (default HTTP/2 idle timeout is typically 60s) and must be re-established
- **Services where upstream connections cycle frequently** — each connection loss forces the next request to block on reconnection, often concurrently across multiple worker threads

## Design

### Overview

The two features are independent and can be configured separately. They are most powerful in combination, but each is useful on its own.

| Feature | What it changes | Where it lives |
|---|---|---|
| `per_upstream_min_connections` | When connections open: proactively, to maintain a floor | `PreconnectPolicy` |
| `connection_aware_load_balancing` | Which host wins host selection: prefer hosts with ready connections | New `Cluster.ConnectionAwareLoadBalancing` message |

### Reference Implementation

Each feature is implemented on its own branch in [bplotnick/envoy](https://github.com/bplotnick/envoy), single commit, against upstream `envoyproxy/envoy:main`:

- **`per_upstream_min_connections`** — [compare view](https://github.com/envoyproxy/envoy/compare/main...bplotnick:envoy:eager-connection-priming)
- **`connection_aware_load_balancing`** — [compare view](https://github.com/envoyproxy/envoy/compare/main...bplotnick:envoy:connection-aware-lb)

### Feature 1: `PreconnectPolicy.per_upstream_min_connections`

#### Configuration

```protobuf
message Cluster {
message PreconnectPolicy {
google.protobuf.DoubleValue per_upstream_preconnect_ratio = 1;
google.protobuf.DoubleValue predictive_preconnect_ratio = 2;

// Minimum open (or opening) connections per upstream host. When non-zero,
// Envoy proactively establishes connections to maintain this floor.
// Default: 0 (disabled — only ratio-based preconnect applies).
google.protobuf.UInt32Value per_upstream_min_connections = 3;
}
}
```

#### Semantics

The invariant is "at least `per_upstream_min_connections` open or opening connections per host on each worker thread." Both established and in-flight connection attempts count toward the floor — this avoids over-priming when multiple events fire concurrently.

#### Triggers

| Event | Action |
|---|---|
| Host added (EDS / DNS / static) | Cluster manager creates the pool for the new host and calls `maybePreconnect()`, which drives the pool up to the floor. |
| Connection closes (GOAWAY, idle timeout, max-duration, max-requests) | The pool itself, inside `ConnPoolImplBase::onConnectionEvent`, calls `tryCreateNewConnections()` after removing the closed client. The floor check lives in `shouldCreateNewConnection` and counts ready + busy + connecting clients ("open or opening") to avoid over-priming under concurrent closes. |

By teaching the pool to honor the floor in its existing preconnect machinery, the per-close trigger falls out for free — every close event already runs through `onConnectionEvent`, so we don't need a separate notification path.

#### Safeguards
- Only healthy hosts are primed (`host->coarseHealth() == Healthy`)
- Circuit breaker limits are checked (`host->canCreateConnection()`)
- A per-worker burst cap (currently 10 concurrent priming attempts) prevents thundering-herd connection storms on large EDS updates; excess hosts queue and drain as attempts complete
- All priming is async via `dispatcher.post()` — never blocks the EDS update path

#### Stats
- `upstream_cx_eager_primed` (counter)
- `upstream_cx_eager_primed_failed` (counter)
- `upstream_cx_eager_pending` (gauge, current queue depth)

Re-priming on connection close shows up in the standard `upstream_cx_total` / `upstream_cx_connect_*` counters — those connections are created through the same pool path as any other, so no separate counter is needed.

### Feature 2: `Cluster.connection_aware_load_balancing`

#### Configuration

```protobuf
message Cluster {
message ConnectionAwareLoadBalancing {
bool enabled = 1;
}
ConnectionAwareLoadBalancing connection_aware_load_balancing = 61;
}
```

#### Semantics

When enabled, `ClusterEntry::chooseHost()` evaluates the host returned by the underlying load balancer. If that host has no ready connection on the current worker, the cluster:

1. **Stimulates connection establishment** to the originally-chosen host (so subsequent requests for it will find it ready). The LB's choice expresses an operator-intended distribution; rather than silently routing elsewhere forever, we use the LB's choice as a signal that this host should be warmed up.
2. Performs up to 3 re-picks looking for a host that is already ready.
3. Falls back to the original selection if no ready host is found (prevents starvation during cold start).

The stimulation step is what makes the feature self-healing as a standalone option. Each "miss" both serves the current request from a ready host *and* warms the originally-chosen host for next time. Within a few requests, the LB's full host set becomes warm, and the configured distribution is restored.

A host has a "ready connection" when at least one of its connection pools on this worker contains a connection that can accept a new stream **without** creating a new connection — i.e. a connection that exists and is not at its concurrent-stream limit. This is exactly Envoy's existing internal `ready_clients_` set inside `ConnPoolImplBase`.

The distinction matters: a host-level `cx_active > 0` check would be insufficient, because an HTTP/2 connection at `max_concurrent_streams` (or an HTTP/1.1 connection currently serving a request) is "active" but useless for a new request. Treating it as "ready" would cause connection-aware LB to leave the host in rotation when it actually has no usable capacity — exactly the failure mode the feature is trying to prevent.

This requires a small interface addition: `Http::ConnectionPool::Instance::hasReadyConnection()`, returning `!ready_clients_.empty()` from the pool's existing internal state. The host-level helper aggregates across all of the host's pools on the worker.

#### Re-pick fallback

The 3-attempt cap and fallback are important: without them, the re-pick loop could starve when no host has yet warmed up (cold start, all hosts simultaneously cycling). Falling back to the LB's original pick means the worst case degrades to current Envoy behavior — never worse.

#### Applicability

This feature is only meaningful for non-consistent-hash load balancing policies (`ROUND_ROBIN`, `LEAST_REQUEST`, `RANDOM`). For consistent-hash policies (`RING_HASH`, `MAGLEV`), re-picks would violate hash affinity, so the setting has no effect.

#### Stats
- `upstream_cx_lb_stimulated` (counter): connection-establishment attempts triggered by an LB miss on the originally-chosen host
- `upstream_cx_lb_repicked` (counter): re-picks that successfully landed on a ready host

### How They Interact

The features are independent and each is safe to use standalone:

- **`min_connections` alone** delivers a guaranteed connection floor up front. Right for latency-sensitive workloads where even a single first-request handshake is unacceptable, or where you want connections established before any traffic arrives.
- **Connection-aware LB alone** delivers lazy preconnect via traffic. Connections get established as the LB tries to use hosts; the stimulation-on-miss behavior prevents traffic concentration. Right for clusters where the operator wants automatic adaptation without configuring an explicit floor.
- **Together**, `min_connections` ensures hosts have connections from the start, and connection-aware LB ensures requests preferentially route to those that are fully ready while continuing to stimulate any hosts that drift below the floor.