envoyproxy / envoyproxy/envoy

coalesce_lb_rebuilds_on_batch_update: Workers can snapshot stale factory state causing RING_HASH ring divergence until next rebalance

Open
#45,055 6 comments 2 reactions 0 assignees View on GitHub
area/load balancing bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

*Description*:
When a health check fails and then a retry instantly passes, we would expect the ring hash to recover back to a healthy state with all the connections intact. However, some pods in our cluster will have some connections considered degraded and traffic will stop being sent on them. In our case, the connections eventually terminated due to an idle timeout. Metrics such as `envoy.cluster.membership_healthy` do not change, and show that all of our connections are being detected healthy.

When the ring next rebalances (due to addition or deletion of a downstream host), all the missing connections are rebuilt and we get our expected number of connections back again.

Disabling `coalesce_lb_rebuilds_on_batch_update` removed this failure behavior, and the suspected root cause is that enabling this feature allows a race condition when our health-check failure and recovery get coalesced into one update, and the overall update is keeping the unhealthy state as the final state.

*Repro steps*:
Conditions under which it reproduced for us reliably (by reliably, we saw this once every 1-2 days across tens of clusters, each with ~10-30 pods running Envoy. Each of these envoy usages connects to 5-20 downstream hosts through the load balancer, with each envoy having two connections to each downstream host).

- Cluster: STRICT_DNS, RING_HASH load balancer, ~10–30 pods running envoy, 5-20 downstream hosts
- Active TCP health-checking with `unhealthy_threshold: 1` (a single missed probe flips state).
- ≥ 2 workers per process (`--concurrency 2` or higher).
- A transient health-check failure on an upstream host (e.g., one missed probe followed by recovery within a few seconds).
- `envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update` enabled

*Suspected failure mode* [AI-generated, but seems consistent with what we've seen]:
#43346 reordered when `ThreadAwareLoadBalancerBase::refresh()` runs relative to `ClusterManagerImpl::postThreadLocalClusterUpdate()`. The reorder opens a cross-thread race that can permanently desync a worker's `lb_` from the main thread's factory state.

With`envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update` disabled: main thread runs `ThreadAwareLoadBalancerBase::refresh()` → writes `factory_->per_priority_state_` under writer lock; then `postThreadLocalClusterUpdate` posts to all workers via `tls_.runOnAllThreads`.

With `envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update` enabled: `refresh()` moves to a MemberUpdateCb, which fires *after* the PriorityUpdateCb chain — but `postThreadLocalClusterUpdate` is still registered as a PriorityUpdateCb ([`cluster_manager_impl.cc#L595-L631`](https://github.com/envoyproxy/envoy/blob/44a99ba/source/common/upstream/cluster_manager_impl.cc#L595-L631)). The new main-thread sequence is:

1. `postThreadLocalClusterUpdate` (`tls_.runOnAllThreads` — posts to all workers, async)
2. `LoadBalancerBase::processDirtyPriorities` (recomputes per-priority health/load/panic)
3. `ZoneAwareLoadBalancerBase::processDirtyPriorities` (regenerates locality routing)
4. `ThreadAwareLoadBalancerBase::refresh()` (writes the factory)

**The race.** Step 1 wakes workers via the dispatcher's eventfd and returns; the main thread continues steps 2–3 synchronously. Steps 2–3 can involve a non-trivial amount of work, so a worker woken by step 1 can call `lb_factory_->create()` ([`thread_aware_lb_impl.cc#L239-L249`](https://github.com/envoyproxy/envoy/blob/44a99ba/source/extensions/load_balancing_policies/common/thread_aware_lb_impl.cc#L239-L249)) before main reaches step 4, snapshotting the *previous* event's `factory_->per_priority_state_` into its per-worker `lb_`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.