envoyproxy / envoyproxy/envoy

Envoy is not freeing memory in a meaningful way to the system

Open
#39,006 43 comments 18 reactions 0 assignees View on GitHub
area/perf bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

**If you are reporting *any* crash or *any* potential security issue, *do not*
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.**

*Title*: Envoy is not freeing memory in a meaningful way to the system

*Description*:
Envoy is not returning memory to the OS in a way that is meaningful to a typical user.

Envoy has a variety of ways to trigger a call to release memory, ultimately calling `tcmalloc::MallocExtension::ReleaseMemoryToSystem`. However, this doesn't seem to actually have the desired impact. I am not an expert here so a bit fuzzy, but my understanding is this only impacts `pageheap_unmapped`. However, this memory is still associated with the process and accounted against it.

Below shows a diagram of memory usage over time.
First, I deploy 5000 routes and clusters to put a lotof memory pressure onto envoy. Then I remove them. Later I run `echo 1 |sudo tee /proc/sys/vm/drop_caches`.

![Image](https://github.com/user-attachments/assets/3bff6cce-ea08-4c75-a310-5888aacf8240).

The metrics graphed are

```
envoy_server_memory_heap_size
envoy_server_memory_allocated
container_memory_working_set_bytes
container_memory_rss
```

Kubernetes uses `container_memory_working_set_bytes` ("container") for the accounting in the system, so this is the only one meaningful to Kubernetes users

At the end, /memory shows:
```
{
"allocated": "19570616",
"heap_size": "880803840",
"pageheap_unmapped": "662102016",
"pageheap_free": "32768",
"total_thread_cache": "22663376",
"total_physical_bytes": "231193006"
}
```

This is with tcmalloc, with latest Envoy on amd64.

Bootstrap has

```
"memory_allocator_manager": {
"bytes_to_release": 100000000
},
```

and `tcmalloc.released_by_timer: 126` shows we are triggering the tcmalloc call. I also ran some bpftrace to be 100% sure the `MallocExtension_Internal_ReleaseMemoryToSystem` function is really called.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.