Envoy v1.34.0+ Shows Increased Latency in Benchmarking

Has anyone noticed performance degradation in the latest Envoy versions **v1.34.0** and **v1.34.1**? We recently identified the performance regression in Envoy starting from version **v1.34.0**, based on our benchmarking results. Specifically, we observed an average latency increase of around 15%-25% compared to earlier versions. This issue persists in v1.34.x, while previous versions such as v1.33.x and v1.32.x do not exhibit this degradation.

For example, our benchmark comparing v1.34.0 to v1.33.3 clearly shows this performance drop:

- P50

- P90

Our benchmark setup and summary:

1. We used [fortio](https://github.com/fortio/fortio) as the traffic generator, varying QPS from 1k to 8k.
2. The test scenario evaluated the overhead of Envoy in a sidecar inbound setup: fortio -> envoy -> service.
3. No filters were enabled except for the basic http_connection_manager and router filter; Envoy simply forwarded requests to the service, and the service returned echoed responses.
4. The performance regression was consistently observed in both v1.34.0 and v1.34.1, while all versions prior to v1.34.0 performed as expected.
5. The latency increase is consistent with the CPU metrics shown below, which indicate higher CPU usage starting from v1.34.0:

Does anyone have insights into what might have changed in v1.34.0? I'd appreciate any suggestions.