elastic / elastic/beats

[Filebeat] Provide a way to clean up old registry entries filtering by key regexp

Open
#51,456 3 comments 1 reaction 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

Currently in Filebeat each input is responsible for managing its own entries in the registry, which includes removing stale ones.

Most of the time this works fine, however if inputs are removed without getting a chance to clean up their entries, they can stay in the store for ever.

This can become an issue for deployments where Filebeat is ingesting a large amount of files, Kubernetes autodiscover is an interesting example:
- It generates inputs per container
- When migrating from multiple inputs (one per container) to single input for the node, inputs stop before fully ingesting the file, which can leave their states behind
- Some K8s nodes can have hundreds (OpenShift [defaults to 250](https://docs.redhat.com/en/documentation/openshift_dedicated/4/html/planning_your_environment/osd-limits-scalability))

We need a way to clean up those entries without having to remove the whole registry. Ideally we'd provide a regexp and remove all entries matching the regexp.

Once Filebeat supports it, we need to also add Elastic Agent support to run this clean up process.

I have two very high level ideas of how this could be implemented:
- A global setting that allows to provide a regexp and when the registry is being initialised, all entries matching the regexp are removed
- A sub-command that cleans up entries matching a regexp while Filebeat is not running.

# Related issues
- https://github.com/elastic/beats/issues/46738

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Filebeat's registry lifecycle and the related issue 46738, then compare the proposed initialization setting and offline sub-command approaches. Done means providing regexp-based removal of matching registry entries and adding Elastic Agent support for the cleanup process.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.