elastic / elastic/beats

libbeat pipeline clients do not detect when all events have been acknowledged

Open
#49,755 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

In Beats receivers, and sometimes in standalone Beats, the libbeat queue is stopped before the Beater is signalled to shut down. This means that there is an interval where the queue begins rejecting events from the clients, but the clients themselves (and the inputs that own them) are still active. This will manifest as Publish calls that trigger AddEvent(event, true) on any event listeners, but then afterwards call DroppedOnPublish on any client listeners when the event can't be added to the queue. Because the pipeline client's helper that tracks outstanding events listens for AddEvent but not DroppedOnPublish, when this happens (which it frequently will as long as events are being published at least ~once a second), the client will continue to think the dropped events are unacknowledged even if all real events receive acknowledgments.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the libbeat queue shutdown path through the Beater, then inspect the pipeline client's outstanding-event helper and its AddEvent and DroppedOnPublish listeners. Reproduce shutdown while events are being published and verify that rejected events no longer remain counted as unacknowledged once all real events receive acknowledgments.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.