elastic / elastic/beats

[performance-profiler] Reuse dissect position buffers to cut hot-path allocations

Open Beginner friendly
#53,164 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

## Hot Path
`(*Dissector).extract` and its callers `(*Dissector).Dissect` / `(*Dissector).DissectConvert` in `libbeat/processors/dissect/dissect.go:57-107` and `libbeat/processors/dissect/dissect.go:117-193` allocate a new `[]position` on every call to `extract`, which shows up as a major allocator hotspot in benchmark profiles.

## Profiling Data
**Before:**
```text
go test ./libbeat/processors/dissect -run '^$' -bench '^BenchmarkDissectComplexStackTraceDegradation$' -benchmem -cpuprofile=/tmp/gh-aw/agent/dissect.before.cpu.prof -memprofile=/tmp/gh-aw/agent/dissect.before.mem.prof -count=5

BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1274858 929.2 ns/op 808 B/op 5 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1256325 930.6 ns/op 808 B/op 5 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1290156 941.9 ns/op 808 B/op 5 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1300606 941.5 ns/op 808 B/op 5 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1282668 927.9 ns/op 808 B/op 5 allocs/op
```

`go tool pprof -top -alloc_space /tmp/gh-aw/agent/dissect.before.mem.prof` excerpt:
```text
flat flat% sum% cum cum%
35.57GB 60.07% 60.07% 35.57GB 60.07% github.com/elastic/beats/v7/libbeat/processors/dissect.normalField.Apply
13.04GB 22.02% 82.09% 48.60GB 82.09% github.com/elastic/beats/v7/libbeat/processors/dissect.(*Dissector).resolve
10.59GB 17.89% 100.00% 10.59GB 17.89% github.com/elastic/beats/v7/libbeat/processors/dissect.(*Dissector).extract
```

## Proposed Change
Use a caller-owned positions buffer in `extract` instead of allocating internally, and use a stack-backed fast path for common field counts in callers.

```diff
diff --git a/libbeat/processors/dissect/dissect.go b/libbeat/processors/dissect/dissect.go
@@ -59,8 +59,15 @@ func (d *Dissector) Dissect(s string) (Map, error) {
- positions, err := d.extract(s)
+ var positions positions
+ if len(d.parser.fields) <= 16 {
+ var stackPositions [16]position
+ positions = stackPositions[:len(d.parser.fields)]
+ } else {
+ positions = make([]position, len(d.parser.fields))
+ }
- if err != nil {
+ if err := d.extract(s, positions); err != nil {
return nil, err
}
@@ -100,8 +114,7 @@ func (d *Dissector) Raw() string {
-func (d *Dissector) extract(s string) (positions, error) {
- positions := make([]position, len(d.parser.fields))
+func (d *Dissector) extract(s string, positions positions) error {
@@
- return positions, nil
+ return nil
}
```

## Results
**After:**
```text
go test ./libbeat/processors/dissect -run '^$' -bench '^BenchmarkDissectComplexStackTraceDegradation$' -benchmem -cpuprofile=/tmp/gh-aw/agent/dissect.after.cpu.prof -memprofile=/tmp/gh-aw/agent/dissect.after.mem.prof -count=5

BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1402860 844.8 ns/op 664 B/op 4 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1385682 857.8 ns/op 664 B/op 4 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1364835 855.8 ns/op 664 B/op 4 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1381454 857.0 ns/op 664 B/op 4 allocs/op
BenchmarkDissectComplexStackTraceDegradation/ComplexStackTrace-8-4 1394032 846.2 ns/op 664 B/op 4 allocs/op
```

`go tool pprof -top -alloc_space /tmp/gh-aw/agent/dissect.after.mem.prof` excerpt:
```text
flat flat% sum% cum cum%
39.46GB 74.15% 74.15% 39.46GB 74.15% github.com/elastic/beats/v7/libbeat/processors/dissect.normalField.Apply
13.74GB 25.83% 100.00% 53.20GB 100.00% github.com/elastic/beats/v7/libbeat/processors/dissect.(*Dissector).resolve
```

**Improvement:**
- `ComplexStackTrace-8-4` memory: `808 B/op -> 664 B/op` (**17.8% reduction**)
- `ComplexStackTrace-8-4` allocations: `5 allocs/op -> 4 allocs/op` (**20.0% reduction**)
- `ComplexStackTrace-8-4` time (mean of shown runs): `934.2 ns/op -> 852.3 ns/op` (**8.8% faster**)

This crosses the reporting threshold via memory/allocation improvement on a benchmarked hot path.

## Verification
- `go test ./libbeat/processors/dissect -count=1` passed after the optimization patch in the profiling run.
- Change is internal-only buffer management in dissect parsing; no API or behavior change intended.

## Evidence
- Baseline benchmark command and optimized benchmark command are identical except for code state.
- Before/after outputs are materially different (B/op and allocs/op changed; timing also changed).
- Duplicate check against `/tmp/previous-findings.json`: no matching open issue title for this exact optimization.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/34607864157)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Sep 18, 2026, 2:22 PM UTC

Contributor guide

Open the contributing guide

Research direction

Start in libbeat/processors/dissect/dissect.go, especially Dissector.extract and its Dissect/DissectConvert callers. Run the named BenchmarkDissectComplexStackTraceDegradation benchmark with -benchmem, then run go test ./libbeat/processors/dissect -count=1. Done means the positions buffer is caller-owned, behavior remains unchanged, and the benchmark confirms lower allocations and memory use.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.