Hardecoded checkpoint when registry log file reaches 10M causes performance issues with high number of files being harvested
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
For confirmed bugs, please report:
- Version: 8.3.3 (confirmed on this version, but affects many older and newer versions, including `main`)
- Operating System: Linux
## Steps to Reproduce:
1. Generate a **HUGE** number of small files for Filebeat to harvest. We're talking something between 500 000 ~ 1 000 000 000 files.
2. Configure a `filestream` input to harvest those files.
3. Start Filebeat
Filebeat will "hang" for a long time (some reports vary from 20-ish min to hours) and then start harvesting the files normally.
If the registry folder is monitored, it's possible to notice that checkpoints are being generated at a very high rate.
Changing the hardcoded value: https://github.com/elastic/beats/blob/deb4147715b4d45a7db8f31f9924365a29860697/libbeat/statestore/backend/memlog/memlog.go#L71
Will increase performance.
## Related issues:
Here is a list of some issues that seem to be related to the checkpoint generation, I haven't completely confirmed, and some are quite old, however they seem related.
- https://github.com/elastic/beats/issues/16076
- https://github.com/elastic/beats/issues/27058
Contributor guide
Research direction
Start with libbeat/statestore/backend/memlog/memlog.go at the hardcoded value around line 71, then reproduce the startup behavior using a filestream input and a very large number of small files. Check how checkpoint generation affects registry activity and startup time. Done means the high-file workload no longer spends an excessive time generating checkpoints before harvesting normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100