Graylog2 / Graylog2/graylog2-server

Netflow input worker process stops handling input and starts dropping data

Open
#13,680 2 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

Netflow inputs to continue taking in data and processing it

## Current Behavior

I have three netflow probes running on my network, sending Netflow v9 data to Graylog, configured with a Netflow UDP input. I've increased the worker count to 6 and the receive buffer size to 524288 to see if it made any difference. Typical input looks to average 150 messages a second on the neflow input.

After a shortish period (roughly a day give or take a few hours), it will stop processing data from one of the netflow probes. If I leave it a bit longer it might stop processing data from a second. I can see from tcpdump that the data is still coming. From netstat I can see that one of the 6 sockets on the netflow input port has filled its receive buffer so new data is just being dropped.

If I try to stop the input, it just moves to state 'stopping' and then cannot be restarted. The only fix is to restart the whole of graylog-server.
I've disabled the pipelines attached to that stream in order to rule out any bug in my processing, with no effect. There are no abnormal numbers of unprocessed messages or any other metrics that I've checked.

Probes 1 and 2 are both running nprobe, with the same configuration and outputting the same Netflow v9 fields. probe 3 runs softflowd, also sending v9 records but with less fields.
For info, the fields I'm sending from nprobe are:
`%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %FIRST_SWITCHED %LAST_SWITCHED %DIRECTION %SRC_VLAN %DST_VLAN %TCP_FLAGS`

I'm guessing that something in the data that occurs occasionally is causing the netflow input to stall trying to process a record. However I have no way to see what is causing it. Is there anywhere that I can turn on additional logging of the netflow input processing without being swamped with data?

Obviously a loss of log data is not a good thing, either from it randomly stopping processing the data from that source, or from a brief outage while I restart the whole graylog service.

## Your Environment

* Graylog Version: 4.3.8
* Java Version: openjdk version "17.0.4" 2022-07-19
* Elasticsearch Version: 7.10.2
* MongoDB Version: 4..0.28
* Operating System: Ubuntu 2.004
* Browser version:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.