elastic / elastic/beats

Misbehaving Filebeat inputs can silently deadlock the config manager without triggering a restart

Open
#47,771 2 comments 1 reaction 1 assignee Claimed by @faec View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

If a Filebeat configuration change requires an input to be shut down but the input does not return from its `Run` method after context cancellation ([for example](https://github.com/elastic/beats/issues/47770)), then the config change listener will deadlock. The problem will not be reported: Filebeat will continue making its regular Agent checkins, and will list all new input states as "configuring" but will neither start any of them nor report an unhealthy status, and thus will never be restarted by Agent, preventing recovery. The only way to get it unstuck from this state (besides restarting Agent entirely) is to make another configuration change that affects the same input type, which will trigger a Filebeat restart after the new configuration message times out.

Filebeat should have some finite timeout while waiting for inputs to shut down, and if they do not finish it should report itself as unhealthy.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.