[Filebeat] Clean up Log input registry entries for removed inputs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
When migrating from Log input to Filestream, the registry entries from the Log input are never removed from the registry, even when the Log input is fully removed from the configuration. Keeping the entries in the registry is usually desirable, so users can add/edit/remove input configurations without losing the state of a file, however when all Log input instances are migrated to Filestream, the old entries are not needed any more.
When the [registrar](https://github.com/elastic/beats/tree/main/filebeat/registrar) used by the Log input starts up, it reads all of its entries from the disk store and maintains an in-memory copy of them, during shut down all those entries are written to disk: https://github.com/elastic/beats/blob/d3be9bf1567bebaf08361c194d325c4038b43618/filebeat/registrar/registrar.go#L139-L143
This prevents Filestream's take over mode from truly removing registry entries from the Log input. Even though the take over mode removes them after the migration: https://github.com/elastic/beats/blob/d3be9bf1567bebaf08361c194d325c4038b43618/filebeat/input/filestream/internal/input-logfile/store.go#L481-L487
They get re-written during shutdown. This happens even if there are no active Log inputs because the registrar is started with Filebeat, even if no instances of the Log input are ever started. Registry entries from files that do not have an active input have their `TTL` set to `-2`.
This issue is about finding a way to clean up the disk store once we know the Log input entries are no longer needed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with filebeat/registrar/registrar.go around startup and shutdown, then inspect filebeat/input/filestream/internal/input-logfile/store.go around takeover cleanup. Trace how inactive Log input entries with TTL -2 are loaded and rewritten, and determine where the disk store can be cleaned once Log input entries are no longer needed. Done means migrated entries are not written back during shutdown while retained state remains available for active configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100