Opt-in leader-aware client balancer
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
(authors: @SaranBalaji90 @jamesmthompson @gyuho)
### What would you like to be added?
Two independent changes, proposed as separate PRs so either can land or revert on its own:
**Phase 1 — return the leader's member ID in every v3 `ResponseHeader`:**
```proto
message ResponseHeader {
uint64 cluster_id = 1;
uint64 member_id = 2;
int64 revision = 3;
uint64 raft_term = 4;
uint64 leader_id = 5;
}
```
- `leader_id` is the serving member's current Raft leader view when the header is filled; `0` means unknown or no elected leader. It is advisory: not necessarily the member that committed the request, not a promise it remains leader, and not an atomic snapshot with `raft_term` — not a fencing token.
- The data is already at the fill site: [`header.fillWithoutRevision`](https://github.com/etcd-io/etcd/blob/5b75ac62cf042a185e902530c25fd3d59c095232/server/etcdserver/api/v3rpc/header.go#L47-L56) writes `member_id`/`raft_term` today, and its [`RaftStatusGetter`](https://github.com/etcd-io/etcd/blob/5b75ac62cf042a185e902530c25fd3d59c095232/server/etcdserver/apply/interface.go#L107-L114) already exposes `Leader()`.
- Wire-compatible: old clients ignore the field; new clients treat `0` from old servers as "no hint"; mixed clusters may return either. JSON/gRPC-gateway gains `leaderId` when nonzero. No request flag — unlike `PrevKv`, this is an in-memory integer already returned by `Status`, costing at most 11 bytes per response.
- Open API question for review, before the field becomes permanent: `leader_id` vs a 2-byte `is_leader` bool. The bool suffices for the minimal client algorithm; the ID additionally lets a client with a trustworthy member-to-route map learn the new leader from a follower's response, and lets operators correlate ordinary responses with membership. Phase 1 should benchmark read/watch overhead, since the header also appears on responses the routing saving never touches.
**Phase 2 — an opt-in leader-aware Go client balancer**, disabled by default with native round-robin unchanged:
```go
cfg.ExperimentalLeaderAware = true
```
- Response-driven: a successful mutation qualifies its route only when the response came directly from the reported leader (`member_id == leader_id != 0` on an unambiguous route). No `Status` polling, no timers, no member-to-endpoint map.
- Scope: mutations where leader routing is an optimization or a requirement — `Put`, `DeleteRange`, `Compact`, non-readonly `Txn` (mirroring the server's `IsTxnReadonly` rule), `LeaseGrant`/`LeaseRevoke`, auth administration, `Alarm`, and `MoveLeader` (followers reject it with `ErrNotLeader`, which the retry policy never retries). Reads, read-only transactions, watches, keep-alives, and member-local maintenance stay on round-robin. Watch streams stay distributed (#18815). A first version could stage KV-only routing.
- Fallback: a stale, failed, or ambiguous hint invalidates itself and round-robin resumes; the cost is at most one forwarded or failed attempt. No new retries, no change to retry classification, forwarding, Raft, or watch behavior. The server never learns a request used leader-aware routing; rollback is configuration-only. Known trade-off: concurrent picks against a gray leader can stall until the first completion invalidates the hint, while round-robin would have sent only a share there; the stall is bounded by the client's in-flight maximum.
- Dependency: Phase 2 needs a supported way to retain the logical route of one transport attempt. A prototype wraps `round_robin` via gRPC's [`endpointsharding`](https://pkg.go.dev/google.golang.org/grpc/balancer/endpointsharding), which is explicitly experimental (#15145, grpc/grpc-go#6472). Before merge we must secure a stable grpc-go composition API or own the endpoint `SubConn` state through supported interfaces, without adding resolver republication churn (#21660).
Requested outcome of this issue: agreement on the Phase 1 field contract (including `leader_id` vs `is_leader`), agreement on the two-PR sequencing (Phase 1 first; Phase 2 lands disabled by default once the field contract and grpc-go boundary are resolved), and guidance on the supported grpc-go picked-route boundary.
### Why is this needed?
Raft commits every write through the leader, but a v3 client cannot tell which member that is. [`ResponseHeader`](https://github.com/etcd-io/etcd/blob/5b75ac62cf042a185e902530c25fd3d59c095232/api/etcdserverpb/rpc.proto#L417-L432) returns `cluster_id`, the serving `member_id`, `revision`, and `raft_term` — not the leader; only [`StatusResponse`](https://github.com/etcd-io/etcd/blob/5b75ac62cf042a185e902530c25fd3d59c095232/api/etcdserverpb/rpc.proto#L1181-L1203) exposes it. Under the default round-robin balancer, a write sent to a follower is forwarded to the leader: an extra follower-to-leader payload copy on two out of three writes in a three-voter cluster. A client that wants the leader today has three options, each with a cost:
- poll `Status` on every endpoint -- steady discovery traffic of `clients × endpoints / interval` (0.1 × clients QPS at a 30s interval), even from idle clients with nothing to route;
- learn only from leader-related errors — but a request sent to an old leader can be forwarded successfully after an election, so errors alone never teach the client the new leader;
- stay on round-robin and keep paying the forwarding copy.
This is a longstanding ask: the v2 client shipped [`EndpointSelectionPrioritizeLeader`](https://pkg.go.dev/go.etcd.io/etcd/client/v2#EndpointSelectionMode) for exactly this reason, and #9157 requested the v3 equivalent in 2018. What has changed: the v3 client was rebuilt on gRPC's resolver and balancer APIs, making a native version possible. Related: #14501 (write degradation when traffic lands on a slow follower) and #15918 (locality-aware balancing and cloud cost).
Measured and modeled benefits, from a local three-member E2E prototype:
- **Peer bytes.** Ideal model for payload `S`: leader-aware sends `2S` (the two replication copies Raft requires) vs round-robin's `2S + (2/3)S` -- a 25% reduction. The prototype measures 24.9% fewer peer bytes per 64-KiB PUT (175,203 → 131,575 bytes of `etcd_network_peer_sent_bytes_total`, sent counter summed across members), matching the model.
- **Cost.** When members sit in different Availability Zones, the avoided bytes are the ones cross-AZ transfer pricing charges for, at the published EC2 rate of $0.01/GB on each side of the transfer. The measured 43,628 avoided bytes per 64-KiB PUT give an estimated savings of ~$226/month at 100 PUT/s and ~$2,262/month at 1,000 PUT/s. This assumes the avoided bytes would otherwise cross AZs -- savings are zero when members share an AZ -- and the etcd counter excludes TCP/TLS/ENI framing, so an AWS experiment should confirm the net savings against `DataTransfer-Regional-Bytes` on both the peer and client paths.
- **Failure behavior.** With a gray (paused) follower, leader-aware writes succeeded 101/101 while round-robin succeeded only 67/101. With a paused leader, the client pays one stale-hint attempt, falls back to round-robin, and re-qualifies the new leader from a normal response (200/275 vs 165/275 over the full window).
- **No discovery traffic.** The leader hint rides ordinary responses, so there is no `Status` loop, no refresh knob; an idle client generates zero discovery traffic and pays at most one forwarded or failed attempt after an election.
c.f., #9157 (original 2018 request), #10941 (custom balancer picker TODO), #14501 (slow-follower write degradation), #15918 (locality and cloud cost), #15145 + grpc/grpc-go#6472 (experimental resolver/balancer boundary), #21660 (resolver churn to avoid), #18815 (why watches stay distributed).
Contributor guide
Assessment
This issue has not been assessed yet.