Graylog2 / Graylog2/graylog2-server

Beats input stops receiving messages even if input is running and accepting connections

Open
#10,460 2 comments 1 reaction 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
Input should not stop receiving messages or at least log an error when this happens.

## Current Behavior
Nodes stop receiving messages and any error is not logged. TCP port used by Beats input is responding when connecting with telnet. There are also many open connections to that TCP port established from clients but no log events are received.
All of ProcessBufferProcessors are in "idle" state.
Thread dump attached.
[graylog_threaddump.txt](https://github.com/Graylog2/graylog2-server/files/6335425/graylog_threaddump.txt)

```
{
"ProcessBufferProcessor #0": "idle",
"ProcessBufferProcessor #1": "idle",
"ProcessBufferProcessor #2": "idle",
"ProcessBufferProcessor #3": "idle",
"ProcessBufferProcessor #4": "idle",
"ProcessBufferProcessor #5": "idle",
"ProcessBufferProcessor #6": "idle",
"ProcessBufferProcessor #7": "idle",
"ProcessBufferProcessor #8": "idle",
"ProcessBufferProcessor #9": "idle",
"ProcessBufferProcessor #10": "idle",
"ProcessBufferProcessor #11": "idle",
"ProcessBufferProcessor #12": "idle",
"ProcessBufferProcessor #13": "idle",
"ProcessBufferProcessor #14": "idle",
"ProcessBufferProcessor #15": "idle",
"ProcessBufferProcessor #16": "idle",
"ProcessBufferProcessor #17": "idle",
"ProcessBufferProcessor #18": "idle",
"ProcessBufferProcessor #19": "idle",
"ProcessBufferProcessor #20": "idle",
"ProcessBufferProcessor #21": "idle",
"ProcessBufferProcessor #22": "idle",
"ProcessBufferProcessor #23": "idle",
"ProcessBufferProcessor #24": "idle",
"ProcessBufferProcessor #25": "idle",
"ProcessBufferProcessor #26": "idle",
"ProcessBufferProcessor #27": "idle",
"ProcessBufferProcessor #28": "idle",
"ProcessBufferProcessor #29": "idle",
"ProcessBufferProcessor #30": "idle",
"ProcessBufferProcessor #31": "idle",
"ProcessBufferProcessor #32": "idle",
"ProcessBufferProcessor #33": "idle",
"ProcessBufferProcessor #34": "idle",
"ProcessBufferProcessor #35": "idle"
}
```

## Possible Solution
Situation is always resolved when restarting Graylog service.

## Your Environment

* Graylog Version: 3.3.11
* Java Version: openjdk 1.8.0.282
* Operating System: RHEL7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.