Graylog2 / Graylog2/graylog2-server

Graylog output performance to Elasticsearch drops dramatically when external output is used

Open
#7,823 3 comments 3 reactions 0 assignees View on GitHub
bug performance triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

Enabling an "external" output should not have a dramatic impact to Graylog's ability to output messages to Elasticsearch.

## Current Behavior

Graylog's output performance to Elasticsearch collapses when an "external" output is enabled.
In our scenario/environment:
- the Elasticsearch cluster is tested to have throughput of ~100.000 indexed documents per second.
- Graylog node is tested to have output of ~30.000 messages per second without "external" output.

When GELF UDP output is used the output drops to ~10.000 messages per second. The performance cost is huge and based on cpu utilisation graphs this is a bug which might be caused by threads being blocked.

Elasticsearch indexing speed. First batch without GELF UDP output. Second batch with GELF UDP output
Screenshot 2020-04-03 at 9 44 25

Graylog node's CPU utilisation is lower with GELF UDP output so performance is **not** due to capacity limitations.
Screenshot 2020-04-03 at 9 45 07

## Possible Solution

I think there is some flaw in the internal logic of external output which causes massive performance issues.

## Steps to Reproduce (for bugs)

1. Setup a Graylog environment
2. Test its throughput without "external" output
3. Create GELF UDP output and point it to an address/port which does not respond
4. Assign the output to default stream
5. Test throughput again

## Context

We use Graylog as a platform for log management and we are forwarding a subset of logs to external SIEM. The environment is large and we require fast throughput. We are having throughput issues due this bug, we need to have 2-3 times more graylog nodes than we would without this bug.

I'm happy to provide more information if required.

## Your Environment

* Graylog Version: 3.1.4
(There are no changes documented in later releases which would fix this, but I can provide results with newer versions if required)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.