Graylog2 / Graylog2/graylog2-server

Negative values in pipeline rules throughput

Open
#19,588 8 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

### Expected Behavior
The throughput should always be correct

### Current Behavior
Throughput values from the Pipeline rules toggle from positive to negative

### Steps to Reproduce
1. You need a lot of traffic, we couldn't reproduce, but the customer also said if we need help with reproducing he is there to help.
2. Recording shows the whole problem live
[recording.webm](https://github.com/Graylog2/support/assets/127098748/255884c5-8993-437d-97ed-150ec8163b86)

3. Create a Pipeline rule with a lot of traffic
4. check the throughput values

### Context
Seeing Throughput values from the Pipeline rules toggle from positive to negative - see attached recording.
Only on one pipeline called 'Streamrouting' which was set up 7 years ago and running.
Customer has ~70 pipelines. They checked some of them but only see the negative values only on 'Streamrouting'.

Customer aware first click to navigate on the Pipeline rules page shows wrong values, but the wrong values are displayed constantly.
They toggle from positive to negative every few seconds without any other interaction with the page.

Values are correct on Manage Pipelines >> Pipelines overview page - don't see negative throughput numbers there.

On one screenshot we see ~300 million msg/s under Throughput but only around ~11k coming in
On second screenshot ~300 million msg/s under Throughput but only around ~11k incoming

Code of the rule with millions of messages:
```
rule "continue to next stage"
when
true
then
set_field("continue", true);
remove_field("continue");
end
```

Screenshot showing minus ~385 million msg/s under Throughput
![1](https://github.com/Graylog2/support/assets/127098748/58de7e69-d451-4692-b994-9fd1e1964def)

Screenshot showing ~300 million msg/s under Throughput but only around ~11k coming in
![2](https://github.com/Graylog2/support/assets/127098748/871a88df-0bce-4eb3-a565-b19d0efacd2a)


The issue was discussed in Slack https://graylog.slack.com/archives/C036LC4K744/p1717668058792819 and from there it sounds like a known issue, and no one submitted a bug report until now.

### Customer Environment
Graylog Version: 5.2.7
OpenSearch Version: 7.10.2
MongoDB Version: 6.0.12

(created from Zendesk ticket #570)
gz#570

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.