elastic / elastic/beats

[Filebeat] Clean up Log input registry entries for removed inputs

Open
#46,738 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

When migrating from Log input to Filestream, the registry entries from the Log input are never removed from the registry, even when the Log input is fully removed from the configuration. Keeping the entries in the registry is usually desirable, so users can add/edit/remove input configurations without losing the state of a file, however when all Log input instances are migrated to Filestream, the old entries are not needed any more.

When the [registrar](https://github.com/elastic/beats/tree/main/filebeat/registrar) used by the Log input starts up, it reads all of its entries from the disk store and maintains an in-memory copy of them, during shut down all those entries are written to disk: https://github.com/elastic/beats/blob/d3be9bf1567bebaf08361c194d325c4038b43618/filebeat/registrar/registrar.go#L139-L143

This prevents Filestream's take over mode from truly removing registry entries from the Log input. Even though the take over mode removes them after the migration: https://github.com/elastic/beats/blob/d3be9bf1567bebaf08361c194d325c4038b43618/filebeat/input/filestream/internal/input-logfile/store.go#L481-L487

They get re-written during shutdown. This happens even if there are no active Log inputs because the registrar is started with Filebeat, even if no instances of the Log input are ever started. Registry entries from files that do not have an active input have their `TTL` set to `-2`.

This issue is about finding a way to clean up the disk store once we know the Log input entries are no longer needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with filebeat/registrar/registrar.go around startup and shutdown, then inspect filebeat/input/filestream/internal/input-logfile/store.go around takeover cleanup. Trace how inactive Log input entries with TTL -2 are loaded and rewritten, and determine where the disk store can be cleaned once Log input entries are no longer needed. Done means migrated entries are not written back during shutdown while retained state remains available for active configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.