envoyproxy / envoyproxy/envoy

Feature request: Add overload action to restart Envoy when it gets stuck

Open
#41,492 14 comments 0 reactions 0 assignees View on GitHub
area/overload_manager enhancement help wanted no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

**Feature request**

This feature request proposes adding a new overload manager action that automatically restarts Envoy when the overload condition remains above a configured threshold for a specified duration.

**Reasoning**

The overload manager is typically configured to stop accepting requests once a certain threshold is reached, as shown by the [documentation example](https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/overload_manager/overload_manager).

However, this can lead to situations where **Envoy becomes permanently stuck** and requires a manual restart. One such case is described [here](https://github.com/envoyproxy/envoy/issues/21923#issuecomment-3378633585), where the Fixed Heap memory monitor caused the (perceived) memory usage to remain high, preventing Envoy from recovering from overload even after memory was freed. While this specific case may be considered a bug, it still highlights undesirable behavior in the overload manager.

Other possible scenarios could include:

* A memory leak during request processing that cause memory pressure to exceed the threshold and never decrease.
* Memory fragmentation that gradually reduces usable memory to the point where any new allocations trigger the overload threshold.
* Similar issues could possibly also occur with CPU utilization, for example if a bug causes a thread to busy loop.

The common pattern in the cases is that the only recovery option is restart, yet the overload manager's configured actions explicitly prevent that from happening (e.g. through an OOM kill).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.