envoyproxy / envoyproxy/envoy

c-ares lib version bump from v1.34.5 to v1.34.6 causing higher latency

Open
#42,909 8 comments 0 reactions 0 assignees View on GitHub
area/dependency area/dns bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

*Description*:
We noticed upgrading from Envoy `v1.36.3` to `v1.36.4` is causing higher latency spikes to our traffic where Envoy is used as an Egress Gateway (mostly relying on connect).

envoy cpu usage with version bump:

Image

latency on p50 with version bump:

Image

Our investigation showed that reverting c-ares lib on top of the last 1.36.x but also on top of last HEAD, resolved the issue. The diff is as follow in our custom build:
```
--- a/bazel/repository_locations.bzl
+++ b/bazel/repository_locations.bzl
@@ -265,12 +265,12 @@ REPOSITORY_LOCATIONS_SPEC = dict(
project_name = "c-ares",
project_desc = "C library for asynchronous DNS requests",
project_url = "https://c-ares.haxx.se/",
- version = "1.34.6",
- sha256 = "912dd7cc3b3e8a79c52fd7fb9c0f4ecf0aaa73e45efda880266a2d6e26b84ef5",
+ version = "1.34.5",
+ sha256 = "7d935790e9af081c25c495fd13c2cfcda4792983418e96358ef6e7320ee06346",
strip_prefix = "c-ares-{version}",
urls = ["https://github.com/c-ares/c-ares/releases/download/v{version}/c-ares-{version}.tar.gz"],
use_category = ["dataplane_core", "controlplane"],
- release_date = "2025-12-08",
+ release_date = "2025-04-08",
cpe = "cpe:2.3:a:c-ares_project:c-ares:*",
license = "c-ares",
license_url = "https://github.com/c-ares/c-ares/blob/cares-{underscore_version}/LICENSE.md",
```

this version bump is coming from https://github.com/envoyproxy/envoy/releases/tag/v1.36.4 where it was fixing a CVE:
```
c-ares/https://github.com/advisories/GHSA-jw54-c8rr-pjpq:
Use after free can crash Envoy due to malfunctioning or compromised DNS.
```

see also https://github.com/c-ares/c-ares/releases/tag/v1.34.6

It is hard to locate where the problem is coming from looking at the changelog from c-ares, but this is definitely causing an issue on our Envoy setup.

*Repro steps*:
> Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

>**Note**: The [Envoy_collect tool](https://github.com/envoyproxy/envoy/blob/main/tools/envoy_collect/README.md)
gathers a tarball with debug logs, config and the following admin
endpoints: /stats, /clusters and /server_info. Please note if there are
privacy concerns, sanitize the data prior to sharing the tarball/pasting.

*Admin and Stats Output*:
>Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
[admin endpoint documentation.](https://www.envoyproxy.io/docs/envoy/latest/operations/admin)

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Config*:
>Include the config used to configure Envoy.

*Logs*:
>Include the access logs and the Envoy logs.

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Call Stack*:
> If the Envoy binary is crashing, a call stack is **required**.
Please refer to the [Bazel Stack trace documentation](https://github.com/envoyproxy/envoy/tree/main/bazel#stack-trace-symbol-resolution).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.