[performance-profiler] Optimize memlog LogOperation by reusing JSON encoder in statestore backend
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
## Hot Path
`libbeat/statestore/backend/memlog/diskstore.go:237-283` (`(*diskstore).LogOperation`) rebuilds JSON encoding state on every operation (`newJSONEncoder` + per-call `countWriter`).
## Profiling Data
**Before (baseline):**
Command (same command used before/after):
```bash
go test -run '^$' -bench 'BenchmarkCRUD/memlog/Set$' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.prof -memprofile=/tmp/gh-aw/agent/mem.prof -count=5 ./libbeat/statestore/backend/benchmarks
```
Benchmark output:
```text
BenchmarkCRUD/memlog/Set-4 142494 9171 ns/op 2795 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 140035 9809 ns/op 2797 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 144481 9994 ns/op 2797 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 128596 10218 ns/op 2798 B/op 54 allocs/op
BenchmarkCRUD/memlog/Set-4 143714 9825 ns/op 2797 B/op 54 allocs/op
```
Average baseline: **9803.4 ns/op**, **2796.8 B/op**, **54 allocs/op**.
`go tool pprof -top` on baseline profile shows serialization/construction overhead in this path (e.g. `github.com/elastic/go-structform/gotype.buildFieldFold` cumulative 16.37%).
## Proposed Change
Reuse encoder/counting state per opened log file instead of recreating on every `LogOperation` call.
Relevant diff locations:
- `libbeat/statestore/backend/memlog/diskstore.go:56-60` add `logEnc` and `logSize` fields.
- `libbeat/statestore/backend/memlog/diskstore.go:204-209` initialize reusable encoder in `tryOpenLog`.
- `libbeat/statestore/backend/memlog/diskstore.go:254-271` replace per-call `countWriter`/`newJSONEncoder` with reused state.
- `libbeat/statestore/backend/memlog/util.go:32-35` add `(*countWriter).reset` helper.
## Results
**After:**
Same benchmark command as baseline.
```text
BenchmarkCRUD/memlog/Set-4 213842 6506 ns/op 1563 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 185740 6531 ns/op 1563 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 225909 6762 ns/op 1564 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 223510 6729 ns/op 1564 B/op 22 allocs/op
BenchmarkCRUD/memlog/Set-4 222123 6981 ns/op 1564 B/op 22 allocs/op
```
Average after: **6701.8 ns/op**, **1563.6 B/op**, **22 allocs/op**.
**Improvement:**
- **31.64% faster** (`ns/op`)
- **44.09% lower memory** (`B/op`)
- **59.26% fewer allocations** (`allocs/op`)
## Verification
- `go test ./libbeat/statestore/backend/memlog ./libbeat/statestore/backend/benchmarks` passed.
- Change is behavior-preserving: it reuses encoder/writer state but keeps the same serialized log format and write order (`logAction` then `op`, newline-delimited).
## Evidence
Commands run:
1. Save candidate patch for memlog files.
2. Restore baseline state for memlog files.
3. Run baseline benchmark (`-count=5`) with benchmem + profiles.
4. Reapply patch.
5. Run post-change benchmark with the same command.
6. Run package tests.
Duplicate check:
- `/tmp/previous-findings.json` contains no existing memlog `LogOperation` encoder-reuse finding.
- GitHub issue search for `statestore memlog performance LogOperation` returned no matching open issue.
> [!NOTE]
>
> 🔒 Integrity filter blocked 1 item
>
> The following item were blocked because they don't meet the GitHub integrity level.
>
> - [#20705](https://github.com/elastic/beats/issues/20705) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
>
> To allow these resources, lower `min-integrity` in your GitHub frontmatter:
>
> ```yaml
> tools:
> github:
> min-integrity: approved # merged | approved | unapproved | none
> ```
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/27423875556)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 19, 2026, 3:16 PM UTC
Contributor guide
Research direction
Start with libbeat/statestore/backend/memlog/diskstore.go, especially tryOpenLog and (*diskstore).LogOperation, then inspect libbeat/statestore/backend/memlog/util.go for countWriter. Run the memlog package and benchmark tests before and after the change. Done means the serialized log format and write order remain unchanged while the benchmark shows lower time, memory use, and allocations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100