elastic / elastic/logstash

Metrics for Persistent Queue

Open
#6,105 0 comments 0 reactions 0 assignees View on GitHub
enhancement monitoring
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

With the introduction of the PQ the inputs and filter/outputs will be asynchronously disconnected.

The overall input events per second can be different from the filter/output events per second and the delta between the two will determine whether the queue is growing or shrinking.

The potential input rate will naturally vary.

Before the introduction of the PQ, the input and filter/output rates were linked so any drop in filter/output rate due to data complexity and/or external latency increases caused a drop in the input rate because the inputs would block. Also a natural drop in the input rate would cause a drop in the filter/output rate because the filter/output will block. It was not possible to know if a drop or rise was caused by natural input rate variance or because of filter/output latency changes.

After the introduction of the PQ it will be possible to know which side is responsible for the change - yay and woohoo, but we need to know the overall queue incoming and outgoing rates.

Double check whether this is covered completely or partially by the Queue ReadClient and WriteClient.

Contributor guide

Open the contributing guide

Research direction

Start by examining the Persistent Queue's Queue ReadClient and WriteClient to determine whether they already expose incoming and outgoing rates. Compare their coverage with the required queue-growth signal, and consider the work complete when both rates can be measured reliably and their difference indicates whether the queue is growing or shrinking.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.