envoyproxy / envoyproxy/envoy

least_request LB strategy causes full TPS drop when upstream endpoint VM reboot

Open
#39,737 11 comments 0 reactions 1 assignee Claimed by @tonya11en View on GitHub
area/load balancing bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Version*: *envoy_v1.34.1*

*Envoy cluster configuration*:
```yaml
- name: cluster_abc
connect_timeout: 3s
lb_policy: LEAST_REQUEST
type: EDS
ignore_health_on_host_removal: true
least_request_lb_config:
choice_count: 3

health_checks:
- timeout: 3s
interval: 4s
unhealthy_threshold: 3
healthy_threshold: 2
no_traffic_interval: 4s
reuse_connection: true
http_health_check:
host: *node_id
path: *health_check_path
always_log_health_check_failures: true

outlier_detection:
split_external_local_origin_errors: true
enforcing_consecutive_5xx: 0
enforcing_consecutive_gateway_failure: 100
enforcing_consecutive_local_origin_failure: 100
enforcing_success_rate: 0
enforcing_local_origin_success_rate: 0
enforcing_failure_percentage: 0
enforcing_failure_percentage_local_origin: 0
consecutive_5xx: 700
consecutive_gateway_failure: 500
consecutive_local_origin_failure: 20
base_ejection_time: 20s
max_ejection_time: 40s
max_ejection_percent: 70
interval: 5s

common_lb_config:
healthy_panic_threshold: { value: 0 }

load_assignment:
cluster_name: service_backend
endpoints:
- locality: { region: R001 }
priority: 0
lb_endpoints:
- endpoint: { address: { socket_address: { address: 192.168.1.1, port_value: 2009 } } }
- endpoint: { address: { socket_address: { address: 192.168.1.2, port_value: 2009 } } }
- endpoint: { address: { socket_address: { address: 192.168.1.3, port_value: 2009 } } }

circuit_breakers:
thresholds:
max_connections: 200
max_requests: 3000
per_host_thresholds:
max_connections: 100
```

*Description*:
>Use Envoy as a proxy in front of 3 upstream service endpoints running on virtual machines. The least_request load balancing policy is configured. When one of the backend endpoint is rebooted (not process-killed), the total TPS observed by the proxy drops to almost zero in several seconds, then recover to normal in several seconds. It takes about 20s from tps starts to drop till tps recovered.

>This issue does not happen when the backend process is killed (e.g., kill -9), where only a small TPS dip is observed, and traffic quickly recovers.

*Reproduction Steps*:

1. Deploy Envoy with least_request load balancing policy pointing to 3 upstream endpoints (preferably VMs).
2. Continuously send grpc traffic at 100 concurrency to envoy.
3. Reboot one of the upstream endpoint VM.

Observe traffic behavior on the client and proxy metrics.

*Expected Behavior*:
Envoy should recognize the endpoint as unavailable or failed in a timely manner, and continue routing traffic to the healthy upstreams, maintaining acceptable TPS levels.

*Actual Behavior*:
Envoy TPS droped to 0 and then recovered to normal.

*Analysis*:
1. The active_requests of rebooted backend VM endpoint was 100.
2. Requests are stucked by the rebooted endpoint, leading to near-complete TPS drop across the service.
3. The TPS back to normal after client request timeout in about 20s.
4. Signal kill -19 to the backend point leads to the same result.

As source code [LeastRequestLoadBalancer::unweightedHostPickNChoices](https://github.com/envoyproxy/envoy/blob/main/source/extensions/load_balancing_policies/least_request/least_request_lb.cc) shown, there are chances of picking the same host twice instead of two different ones(also metioned by #11004 and #14859), where causes unacceptable result in my scenario. I also tried FULL_SCAN mode, it won't casue full tps drop, but it is not suitable for scenarios with numerous upstream hosts.

So, is there a better way of avoid picking the same host twice? Or add another option of [LeastRequest.SelectionMethod](https://www.envoyproxy.io/docs/envoy/v1.34.1/api-v3/extensions/load_balancing_policies/least_request/v3/least_request.proto.html#enum-extensions-load-balancing-policies-least-request-v3-leastrequest-selectionmethod) that can pick different hosts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.