elastic / elastic/beats

[Filebeat] Add configurable memlog checkpoint size for the registry

Open
#50,114 3 comments 0 reactions 1 assignee Claimed by @AndersonQ View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

**Context:** This change is being split out from the `growing_fingerprint` work
on #44780 to ship a general-purpose enhancement independently.

### Description

The memlog registry backend triggers a checkpoint (full state snapshot) when the
WAL (write-ahead log) file exceeds an internal threshold. This threshold is
currently hardcoded at 10 MB and cannot be configured by users.

For most workloads, 10 MB is fine. However, at scale (25k+ tracked files) with
file identity strategies that produce larger registry keys (such as the upcoming
`growing_fingerprint`, which stores up to 2000-character keys), the 10 MB
threshold causes excessive checkpointing. Each checkpoint serializes the entire
in-memory store synchronously while holding the store lock, blocking all
operations.

Benchmarks showed that at 25k files with `growing_fingerprint`, the 10 MB
checkpoint caused a **26% drop in EPS** (43k vs 58k). Increasing to 35 MB
restored full throughput (see https://github.com/elastic/beats/issues/44780#issuecomment-4241778332).

### Changes

- Add `libbeat/statestore/backend/memlog/config.go`: new `Config` struct with
`CheckpointSize` field
- Add `filebeat.registry.memlog.checkpoint_size` configuration option
- Wire the config through `filebeat/config/config.go` and
`filebeat/beater/store.go`
- Add unit tests for checkpoint threshold behavior
- Document the new option in the reference YAML configuration

### Configuration

```yaml
filebeat.registry:
memlog:
# Checkpoint threshold in bytes. Default: 10485760 (10 MB).
checkpoint_size: 52428800 # 50 MB
```

### Impact

No behavior change for existing users — the default remains 10 MB. Users who
track a large number of files (25k+) and experience throughput degradation from
checkpoint frequency can now tune this value.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.