elastic / elastic/beats

[performance-profiler] Optimize memlog LogOperation by reusing JSON encoder in statestore backend

Open
#51,218 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

## Hot Path
`libbeat/statestore/backend/memlog/diskstore.go:237-283` (`(*diskstore).LogOperation`) rebuilds JSON encoding state on every operation (`newJSONEncoder` + per-call `countWriter`).

## Profiling Data
**Before (baseline):**
Command (same command used before/after):
```bash
go test -run '^$' -bench 'BenchmarkCRUD/memlog/Set$' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.prof -memprofile=/tmp/gh-aw/agent/mem.prof -count=5 ./libbeat/statestore/backend/benchmarks
```

Benchmark output:
```text
BenchmarkCRUD/memlog/Set-4 142494 9171 ns/op 2795 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 140035 9809 ns/op 2797 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 144481 9994 ns/op 2797 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 128596 10218 ns/op 2798 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 143714 9825 ns/op 2797 B/op 54 allocs/op
```

Average baseline: **9803.4 ns/op**, **2796.8 B/op**, **54 allocs/op**.

`go tool pprof -top` on baseline profile shows serialization/construction overhead in this path (e.g. `github.com/elastic/go-structform/gotype.buildFieldFold` cumulative 16.37%).

## Proposed Change
Reuse encoder/counting state per opened log file instead of recreating on every `LogOperation` call.

Relevant diff locations:
- `libbeat/statestore/backend/memlog/diskstore.go:56-60` add `logEnc` and `logSize` fields.
- `libbeat/statestore/backend/memlog/diskstore.go:204-209` initialize reusable encoder in `tryOpenLog`.
- `libbeat/statestore/backend/memlog/diskstore.go:254-271` replace per-call `countWriter`/`newJSONEncoder` with reused state.
- `libbeat/statestore/backend/memlog/util.go:32-35` add `(*countWriter).reset` helper.

## Results
**After:**
Same benchmark command as baseline.

```text
BenchmarkCRUD/memlog/Set-4 213842 6506 ns/op 1563 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 185740 6531 ns/op 1563 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 225909 6762 ns/op 1564 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 223510 6729 ns/op 1564 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 222123 6981 ns/op 1564 B/op 22 allocs/op
```

Average after: **6701.8 ns/op**, **1563.6 B/op**, **22 allocs/op**.

**Improvement:**
- **31.64% faster** (`ns/op`)
- **44.09% lower memory** (`B/op`)
- **59.26% fewer allocations** (`allocs/op`)

## Verification
- `go test ./libbeat/statestore/backend/memlog ./libbeat/statestore/backend/benchmarks` passed.
- Change is behavior-preserving: it reuses encoder/writer state but keeps the same serialized log format and write order (`logAction` then `op`, newline-delimited).

## Evidence
Commands run:
1. Save candidate patch for memlog files.
2. Restore baseline state for memlog files.
3. Run baseline benchmark (`-count=5`) with benchmem + profiles.
4. Reapply patch.
5. Run post-change benchmark with the same command.
6. Run package tests.

Duplicate check:
- `/tmp/previous-findings.json` contains no existing memlog `LogOperation` encoder-reuse finding.
- GitHub issue search for `statestore memlog performance LogOperation` returned no matching open issue.

> [!NOTE]
>
> 🔒 Integrity filter blocked 1 item
>
> The following item were blocked because they don't meet the GitHub integrity level.
>
> - [#20705](https://github.com/elastic/beats/issues/20705) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
>
> To allow these resources, lower `min-integrity` in your GitHub frontmatter:
>
> ```yaml
> tools:
> github:
> min-integrity: approved # merged | approved | unapproved | none
> ```
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/27423875556)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 19, 2026, 3:16 PM UTC

Contributor guide

Open the contributing guide

Research direction

Start with libbeat/statestore/backend/memlog/diskstore.go, especially tryOpenLog and (*diskstore).LogOperation, then inspect libbeat/statestore/backend/memlog/util.go for countWriter. Run the memlog package and benchmark tests before and after the change. Done means the serialized log format and write order remain unchanged while the benchmark shows lower time, memory use, and allocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.