Graylog2 / Graylog2/graylog2-server

ES Index API is unreachable even if there is reachable hosts in elasticsearch_hosts variable

Open
#7,887 3 comments 2 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
Graylog should skip unreachable hosts defined in elasticsearch_hosts variable and try to call API of other hosts instead.

## Current Behavior

Two hosts in elasticsearch_hosts variable, eg.:
`elasticsearch_hosts = http://es-coord001.example.com:9200,http://es-coord002.example.com:9200`

If one of the hosts defined in elasticsearch_hosts is down (in our case it was first one on the list), there comes following errors in to server.log:

`2020-04-15T00:00:02.669+03:00 ERROR [IndexFieldTypePoller] Couldn't get mapping for index : No route to host (Host unreachable).`

`2020-04-15T00:00:02.669+03:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index

Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)`

`2020-04-15T00:00:05.675+03:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set
….
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)`

`2020-04-15T00:00:26.717+03:00 ERROR [Messages] Caught exception during bulk indexing: java.net.NoRouteToHostException: No route to host (Host unreachable), retrying (attempt #1).`

Also System > Indices -page in UI gives the following errors and do not list any index sets:

![ui_error](https://user-images.githubusercontent.com/41315949/79414714-00a6e580-7fb4-11ea-9ffc-3ff3bf2db184.png)

Tried also to change unresponding host in elasticsearch_hosts something not even found in DNS and that host was skipped. So problem exists if DNS entry for the host exists but host is not responding to port 9200.

## Possible Solution

Unresponding hosts should be skipped just like the ones with missing DNS entry.

## Steps to Reproduce (for bugs)
1. Set for example two hosts to Graylog’s config variable elasticsearch_hosts and ensure that page System > Indices is listing all the index sets.
2. Shut down either one of the hosts and check the System > Indices page again.

## Context

Situation was noticed when one or our ES coordinating nodes defined in elasticsearch_hosts was down for a maintenance and I tried to open the System > Indices page during that.

## Your Environment

* Graylog Version: 3.2.4
* Elasticsearch Version: 6.8.8
* MongoDB Version: 4.0.17
* Operating System: RHEL 7.7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.