elastic / elastic/beats

[performance-profiler] Avoid rescanning acknowledged memqueue entries for byte metrics

Open
#52,016 1 comment 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

## Finding
`(*runLoop[T]).handleDelete` in `libbeat/publisher/queue/memqueue/runloop.go:220-231` performs a second ring-buffer traversal to sum `eventSize` for `RemoveEvents`. The same batch was already traversed and summed by `handleGetReply` at `runloop.go:208-217`; the byte total can be carried with the batch acknowledgment.

## Evidence
Scoped command used for both isolated baseline and candidate copies (the temporary benchmark harness exercised the existing package function directly):
```text
go test -run='^$' -bench='^BenchmarkDeleteEvents$' -benchmem -benchtime=3s -count=5 -cpu=1 ./libbeat/publisher/queue/memqueue
```

Baseline (`handleDelete(count)` rescans 10,000 entries):
```text
31585, 29685, 33269, 29057, 26302 ns/op; 0 B/op; 0 allocs/op
```
Candidate (batch carries the already-computed byte total; no rescan):
```text
5.393, 8.349, 8.076, 4.969, 5.624 ns/op; 0 B/op; 0 allocs/op
```
Median: `29,685 -> 5.624 ns/op` (**99.98% faster**). CPU profiles showed baseline `handleDelete` at 100% of samples; candidate retained only 64.8% in `handleDelete` with the benchmark observer accounting for the remainder.

## Proposed fix
Add the byte total to `batch`, aggregate it in the existing `ackLoop.handleBatchSig` batch-list traversal (`ackloop.go:66-78`), send `{count, bytes}` through `deleteChan`, and make `handleDelete` update counters and `RemoveEvents` without reading the ring buffer. Preserve the existing `ConsumeEvents`/`RemoveEvents` byte values, including cancelled batches.

## Verification
`go test ./libbeat/publisher/queue/memqueue` passed in the candidate copy, including the observer metric tests. The shared repository was not modified.

## Duplicate check
No duplicate found among the supplied open performance issues. #49629 targets diskqueue `reader_loop.go`; #51047/#51680 concern slabqueue design/throughput; the listed memqueue/channel-pooling issue is distinct from this acknowledged-entry rescan. Recent memqueue history (`#50976`, `#51047`, `#51136`) does not contain this optimization.

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `169.254.169.254`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "169.254.169.254"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/29507431266)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jul 23, 2026, 2:56 PM UTC

Contributor guide

Open the contributing guide

Research direction

Start with handleGetReply and handleDelete in libbeat/publisher/queue/memqueue/runloop.go, then read ackLoop.handleBatchSig in ackloop.go to understand how acknowledged batches are aggregated. Run the provided BenchmarkDeleteEvents command and the memqueue package tests. Done means the ring-buffer rescan is removed while ConsumeEvents and RemoveEvents byte metrics remain unchanged, including cancelled batches.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.