[performance-profiler] Avoid rescanning acknowledged memqueue entries for byte metrics
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Finding
`(*runLoop[T]).handleDelete` in `libbeat/publisher/queue/memqueue/runloop.go:220-231` performs a second ring-buffer traversal to sum `eventSize` for `RemoveEvents`. The same batch was already traversed and summed by `handleGetReply` at `runloop.go:208-217`; the byte total can be carried with the batch acknowledgment.
## Evidence
Scoped command used for both isolated baseline and candidate copies (the temporary benchmark harness exercised the existing package function directly):
```text
go test -run='^$' -bench='^BenchmarkDeleteEvents$' -benchmem -benchtime=3s -count=5 -cpu=1 ./libbeat/publisher/queue/memqueue
```
Baseline (`handleDelete(count)` rescans 10,000 entries):
```text
31585, 29685, 33269, 29057, 26302 ns/op; 0 B/op; 0 allocs/op
```
Candidate (batch carries the already-computed byte total; no rescan):
```text
5.393, 8.349, 8.076, 4.969, 5.624 ns/op; 0 B/op; 0 allocs/op
```
Median: `29,685 -> 5.624 ns/op` (**99.98% faster**). CPU profiles showed baseline `handleDelete` at 100% of samples; candidate retained only 64.8% in `handleDelete` with the benchmark observer accounting for the remainder.
## Proposed fix
Add the byte total to `batch`, aggregate it in the existing `ackLoop.handleBatchSig` batch-list traversal (`ackloop.go:66-78`), send `{count, bytes}` through `deleteChan`, and make `handleDelete` update counters and `RemoveEvents` without reading the ring buffer. Preserve the existing `ConsumeEvents`/`RemoveEvents` byte values, including cancelled batches.
## Verification
`go test ./libbeat/publisher/queue/memqueue` passed in the candidate copy, including the observer metric tests. The shared repository was not modified.
## Duplicate check
No duplicate found among the supplied open performance issues. #49629 targets diskqueue `reader_loop.go`; #51047/#51680 concern slabqueue design/throughput; the listed memqueue/channel-pooling issue is distinct from this acknowledged-entry rescan. Recent memqueue history (`#50976`, `#51047`, `#51136`) does not contain this optimization.
> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `169.254.169.254`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "169.254.169.254"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/29507431266)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jul 23, 2026, 2:56 PM UTC
Contributor guide
Research direction
Start with handleGetReply and handleDelete in libbeat/publisher/queue/memqueue/runloop.go, then read ackLoop.handleBatchSig in ackloop.go to understand how acknowledged batches are aggregated. Run the provided BenchmarkDeleteEvents command and the memqueue package tests. Done means the ring-buffer rescan is removed while ConsumeEvents and RemoveEvents byte metrics remain unchanged, including cancelled batches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100