elastic / elastic/beats

filebeat (practically) hangs after restart on machine with a lot of logs

Open
#16,076 20 comments 0 reactions 0 assignees View on GitHub
Filebeat Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

_Sorry for somewhat vague problem description. I was working on the problem in panic mode and had no time to preserve details like exact log snippets._

I use filebeat to scrap logs from machines where i have PLENTY of log files. Numbers vary, but having a few thousands new files every day is normal and sth like 50 thousands happens (in general: a lot of small apps running, each app opens new log on each restart and after each midnight, old logs are moved out by name, log names contain date…). After some initial tuning (harvester limit adapted to system capacity, time limits tuned to abandon inactive files at reasonable rate, etc) filebeat handles those setups without problems.

Well, handled until I reconfigured. Today I split some inputs into a few sections, more or less it was change from
```
- type: log
enabled: true
paths:
- /some/path/*.log
- /other/path/*.log
- …
# and other settings
```
into
```
- type: log
enabled: true
paths:
- /some/path/*.log
# and other settings
- type: log
enabled: true
paths:
- /other/path/*.log
# and other settings
- type: log
enabled: true
paths:
- …remaining …
# and other settings
```
where „other settings” were the same everywhere except I added some custom fields in each section (this was the reason for the change), and reduced harvester limits so they summed up to the original value.

After restart, filebeat practically stopped working. It either didn't forward anything, or handled only one section (depending on the machine, I saw the problem on a few). After I enabled debug logging it only logged myriads of notes about reading and writing data.json with some statistics („XYZV before, XYZV after”) and later started to put some notes about finding or adding state of individual files (~ 1 such note per second). On the machine I fixed last it remained in such a state for about 2 hours, without noticeable change.

I recovered from that by stopping filebeat, erasing ``/var/lib/filebeat/registry``, and starting filebeat again – then it simply started to work and seems to work properly (of course it republished all old logs according to my age settings and introduced lag caused by old data processing).

I suppose there is sth wrong in the way state is migrated/interpreted/checked after input sections change. This is somewhat unexpected, I didn't think that changing alocation of files to sections impacts state handling.

Tested on filebeat 7.5.1

PS My ``/var/lib/filebeat/registry/data.json`` have 3-8MB, if that is of any interest.

Contributor guide

Open the contributing guide

Research direction

Start with the restart and input-splitting scenario described for Filebeat 7.5.1, using /var/lib/filebeat/registry/data.json and debug logs as the main evidence. Reproduce with many rotated log files and compare behavior before and after changing input sections; done means Filebeat resumes forwarding without hours of registry activity or requiring deletion of the registry.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.