[Filebeat] Provide a way to clean up old registry entries filtering by key regexp
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
Currently in Filebeat each input is responsible for managing its own entries in the registry, which includes removing stale ones.
Most of the time this works fine, however if inputs are removed without getting a chance to clean up their entries, they can stay in the store for ever.
This can become an issue for deployments where Filebeat is ingesting a large amount of files, Kubernetes autodiscover is an interesting example:
- It generates inputs per container
- When migrating from multiple inputs (one per container) to single input for the node, inputs stop before fully ingesting the file, which can leave their states behind
- Some K8s nodes can have hundreds (OpenShift [defaults to 250](https://docs.redhat.com/en/documentation/openshift_dedicated/4/html/planning_your_environment/osd-limits-scalability))
We need a way to clean up those entries without having to remove the whole registry. Ideally we'd provide a regexp and remove all entries matching the regexp.
Once Filebeat supports it, we need to also add Elastic Agent support to run this clean up process.
I have two very high level ideas of how this could be implemented:
- A global setting that allows to provide a regexp and when the registry is being initialised, all entries matching the regexp are removed
- A sub-command that cleans up entries matching a regexp while Filebeat is not running.
# Related issues
- https://github.com/elastic/beats/issues/46738
Contributor guide
Research direction
Start by reviewing Filebeat's registry lifecycle and the related issue 46738, then compare the proposed initialization setting and offline sub-command approaches. Done means providing regexp-based removal of matching registry entries and adding Elastic Agent support for the cleanup process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100