Graylog2 / Graylog2/graylog2-server

Graylog doesn't process messages under certain conditions

Open
#7,906 1 comment 0 reactions 0 assignees View on GitHub
to-verify
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Current setup
I have 56 graylog nodes processing logs from several data centers to elasticsearch.
Elasticsearch cluster consist of 360 data nodes, several master nodes and 60 coordinator nodes.

Graylog is configured to use service discovery and to read\write only to coordinator nodes for better load distribution in elasticsearch.
The typical workload is 50k-200k messages per second with peaks up to 1-2m messages per second.

elasticsearch related config of graylog (tuned a bit for big indices and long-running queries):

elasticsearch_discovery_enabled = true
elasticsearch_discovery_filter = type:coordinator
elasticsearch_disable_version_check = false
elasticsearch_connect_timeout = 60s
elasticsearch_max_total_connections = 600
elasticsearch_max_total_connections_per_route = 50
elasticsearch_max_retries = 10
elasticsearch_index_optimization_jobs = 360
elasticsearch_discovery_frequency = 5s
elasticsearch_socket_timeout = 300s
elasticsearch_analyzer = standard
elasticsearch_request_timeout = 15m
elasticsearch_index_optimization_timeout = 4h

## Expected Behavior
If a data center with data nodes goes down, elasticsearch elects a new master and promotes replica shards to primary.
This is happening with a small delay due to https://github.com/elastic/elasticsearch/issues/46909.
While the elasticsearch master is performing it's job graylog nodes are writing logs to the journal.
When shards are promoted to primary graylog starts writing data through coordinators into elasticsearch.

## Current Behavior
Just the same as in the "expected behavior", but graylog nodes aren't writing anything to elasticsearch.
Restarting any graylog node fixes the problem for the node. It starts writing logs to elastic while others are still writing to the journal. So there isn't really a elasticsearch issue - no errors in logs, no dead threads, nothing suspicious.
If I perform a restart on all coordinator nodes then the overwhelmed graylog nodes start to write logs again - all at once.
No errors in logs, nothing useful in debug messages.

We faced the problem after updating to 3.2

## Steps to Reproduce
1. create an elasticsearch cluster consisting of coordinator nodes (no data on these), data nodes and masters. (at least 20-30 hosts)
2. tag them as masters, coordinators and data, create an index (1 replica shard to each primary. the number of primary shards is half of the amount of data nodes present)
3. set cluster.routing.allocation.same_shard.host: true on data nodes
4. configure graylog for service discovery and apply a filter to get only coordinator nodes.
3. start indexing logs with graylog (fill an index with at least 10-20gb of data)
4. stop half of data nodes simultanously while indexing (cluster will degrade to yellow state)
5. see journals still filling up on graylog when shards are already promoted
6. restart a single graylog node and in will start processing logs
7. restart all coordinator nodes, all graylog nodes will start processing logs to yellow cluster

## Environment
* Graylog Version: 3.2.4
* Elasticsearch Version: 6.8.8
* MongoDB Version: 4.0.8
* Operating System: CentOS7
* Browser version: any

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.