Graylog2 / Graylog2/graylog2-server
Graylog output performance to Elasticsearch drops dramatically when external output is used
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
Enabling an "external" output should not have a dramatic impact to Graylog's ability to output messages to Elasticsearch.
## Current Behavior
Graylog's output performance to Elasticsearch collapses when an "external" output is enabled.
In our scenario/environment:
- the Elasticsearch cluster is tested to have throughput of ~100.000 indexed documents per second.
- Graylog node is tested to have output of ~30.000 messages per second without "external" output.
When GELF UDP output is used the output drops to ~10.000 messages per second. The performance cost is huge and based on cpu utilisation graphs this is a bug which might be caused by threads being blocked.
Elasticsearch indexing speed. First batch without GELF UDP output. Second batch with GELF UDP output

Graylog node's CPU utilisation is lower with GELF UDP output so performance is **not** due to capacity limitations.

## Possible Solution
I think there is some flaw in the internal logic of external output which causes massive performance issues.
## Steps to Reproduce (for bugs)
1. Setup a Graylog environment
2. Test its throughput without "external" output
3. Create GELF UDP output and point it to an address/port which does not respond
4. Assign the output to default stream
5. Test throughput again
## Context
We use Graylog as a platform for log management and we are forwarding a subset of logs to external SIEM. The environment is large and we require fast throughput. We are having throughput issues due this bug, we need to have 2-3 times more graylog nodes than we would without this bug.
I'm happy to provide more information if required.
## Your Environment
* Graylog Version: 3.1.4
(There are no changes documented in later releases which would fix this, but I can provide results with newer versions if required)
Contributor guide
Assessment
This issue has not been assessed yet.