envoyproxy / envoyproxy/envoy

Improve outbound connection balancing to balance across virtual listeners

Open
#21,575 4 comments 0 reactions 0 assignees View on GitHub
area/listener help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

Title: Improve outbound connection balancing to balance across virtual listeners

Description:
We have a usecase with a set of hosts where each host maintains a single pair of long lived TCP connections (one inbound and one outbound) to each other host. All traffic between any pair of hosts is multiplexed over this single pair of connections. We're seeing problems with outbound connections from a given host not being balanced across Envoy workers even when we configure exact connection balancing on the outbound side. For example, with a 10 host configuration (9 outbound connections per host) with 6 workers, and exact connection balancing configured, one of the hosts shows the following distribution of outbound connections across workers:

envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.165.145_27010"} 1
envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.169.99_27010"} 1
envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.172.109_27010"} 1
envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.176.127_27010"} 1
envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.176.177_27010"} 1
envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.179.17_27010"} 1
envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.188.61_27010"} 1
envoy_listener_worker_4_downstream_cx_total{listener_address="10.10.160.89_27010"} 1
envoy_listener_worker_4_downstream_cx_total{listener_address="10.10.181.214_27010"} 1

As far as I can tell, the reason the connection balancing isn't working better in this case is that we have 9 distinct virtual listeners (one per destination IP) and the connection balancer is only balancing across workers *within* a given virtual listener. Since we only have a single outbound connection per virtual listener/destination there really isn't any useful balancing that can be done within that virtual listener. What would work better in our case (or it seems any case with a very small number of long lived connections per destination) would be to balance outbound connections across workers globally within the Envoy instance (not on a per listener basis).

To test this theory I hacked some changes to increment a global sequence number every time an outbound connection is handed off to the per destination listener, and then select the worker number for a given connection on a global basis based on sequenceNum % workerCount. Code is [here](https://github.com/drorke/envoy/commit/945b618a7b7597c6b12ecfc1f33406d1ad14bba9). This is a really ugly hack and I'm not suggesting it's the right design, but it did seem to give me much better outbound connection balancing:

envoy_listener_worker_0_downstream_cx_active{listener_address="10.10.238.59_27010"} 1
envoy_listener_worker_0_downstream_cx_active{listener_address="10.10.243.169_27010"} 1
envoy_listener_worker_1_downstream_cx_active{listener_address="10.10.234.233_27010"} 1
envoy_listener_worker_1_downstream_cx_active{listener_address="10.10.247.215_27010"} 1
envoy_listener_worker_2_downstream_cx_active{listener_address="10.10.230.35_27010"} 1
envoy_listener_worker_3_downstream_cx_active{listener_address="10.10.228.246_27010"} 1
envoy_listener_worker_4_downstream_cx_active{listener_address="10.10.232.143_27010"} 1
envoy_listener_worker_5_downstream_cx_active{listener_address="10.10.233.95_27010"} 1
envoy_listener_worker_5_downstream_cx_active{listener_address="10.10.255.143_27010"} 1

Can we consider enhancing the outbound connection balancing in some way so that it balances with a more global scope and not just within a given virtual listener/destination?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.