Envoy is not freeing memory in a meaningful way to the system
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 428
Description
**If you are reporting *any* crash or *any* potential security issue, *do not*
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.**
*Title*: Envoy is not freeing memory in a meaningful way to the system
*Description*:
Envoy is not returning memory to the OS in a way that is meaningful to a typical user.
Envoy has a variety of ways to trigger a call to release memory, ultimately calling `tcmalloc::MallocExtension::ReleaseMemoryToSystem`. However, this doesn't seem to actually have the desired impact. I am not an expert here so a bit fuzzy, but my understanding is this only impacts `pageheap_unmapped`. However, this memory is still associated with the process and accounted against it.
Below shows a diagram of memory usage over time.
First, I deploy 5000 routes and clusters to put a lotof memory pressure onto envoy. Then I remove them. Later I run `echo 1 |sudo tee /proc/sys/vm/drop_caches`.
.
The metrics graphed are
```
envoy_server_memory_heap_size
envoy_server_memory_allocated
container_memory_working_set_bytes
container_memory_rss
```
Kubernetes uses `container_memory_working_set_bytes` ("container") for the accounting in the system, so this is the only one meaningful to Kubernetes users
At the end, /memory shows:
```
{
"allocated": "19570616",
"heap_size": "880803840",
"pageheap_unmapped": "662102016",
"pageheap_free": "32768",
"total_thread_cache": "22663376",
"total_physical_bytes": "231193006"
}
```
This is with tcmalloc, with latest Envoy on amd64.
Bootstrap has
```
"memory_allocator_manager": {
"bytes_to_release": 100000000
},
```
and `tcmalloc.released_by_timer: 126` shows we are triggering the tcmalloc call. I also ran some bpftrace to be 100% sure the `MallocExtension_Internal_ReleaseMemoryToSystem` function is really called.
Contributor guide
Assessment
This issue has not been assessed yet.