Misbehaving Filebeat inputs can silently deadlock the config manager without triggering a restart
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
If a Filebeat configuration change requires an input to be shut down but the input does not return from its `Run` method after context cancellation ([for example](https://github.com/elastic/beats/issues/47770)), then the config change listener will deadlock. The problem will not be reported: Filebeat will continue making its regular Agent checkins, and will list all new input states as "configuring" but will neither start any of them nor report an unhealthy status, and thus will never be restarted by Agent, preventing recovery. The only way to get it unstuck from this state (besides restarting Agent entirely) is to make another configuration change that affects the same input type, which will trigger a Filebeat restart after the new configuration message times out.
Filebeat should have some finite timeout while waiting for inputs to shut down, and if they do not finish it should report itself as unhealthy.
Contributor guide
Assessment
This issue has not been assessed yet.