elastic / elastic/beats

Hardecoded checkpoint when registry log file reaches 10M causes performance issues with high number of files being harvested

Open
#33,382 9 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

For confirmed bugs, please report:
- Version: 8.3.3 (confirmed on this version, but affects many older and newer versions, including `main`)
- Operating System: Linux

## Steps to Reproduce:
1. Generate a **HUGE** number of small files for Filebeat to harvest. We're talking something between 500 000 ~ 1 000 000 000 files.
2. Configure a `filestream` input to harvest those files.
3. Start Filebeat

Filebeat will "hang" for a long time (some reports vary from 20-ish min to hours) and then start harvesting the files normally.

If the registry folder is monitored, it's possible to notice that checkpoints are being generated at a very high rate.

Changing the hardcoded value: https://github.com/elastic/beats/blob/deb4147715b4d45a7db8f31f9924365a29860697/libbeat/statestore/backend/memlog/memlog.go#L71
Will increase performance.

## Related issues:
Here is a list of some issues that seem to be related to the checkpoint generation, I haven't completely confirmed, and some are quite old, however they seem related.
- https://github.com/elastic/beats/issues/16076
- https://github.com/elastic/beats/issues/27058

Contributor guide

Open the contributing guide

Research direction

Start with libbeat/statestore/backend/memlog/memlog.go at the hardcoded value around line 71, then reproduce the startup behavior using a filestream input and a very large number of small files. Check how checkpoint generation affects registry activity and startup time. Done means the high-file workload no longer spends an excessive time generating checkpoints before harvesting normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.