Graylog2 / Graylog2/graylog2-server
ES Index API is unreachable even if there is reachable hosts in elasticsearch_hosts variable
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
Graylog should skip unreachable hosts defined in elasticsearch_hosts variable and try to call API of other hosts instead.
## Current Behavior
Two hosts in elasticsearch_hosts variable, eg.:
`elasticsearch_hosts = http://es-coord001.example.com:9200,http://es-coord002.example.com:9200`
If one of the hosts defined in elasticsearch_hosts is down (in our case it was first one on the list), there comes following errors in to server.log:
`2020-04-15T00:00:02.669+03:00 ERROR [IndexFieldTypePoller] Couldn't get mapping for index : No route to host (Host unreachable).`
`2020-04-15T00:00:02.669+03:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
…
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)`
`2020-04-15T00:00:05.675+03:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set
….
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)`
`2020-04-15T00:00:26.717+03:00 ERROR [Messages] Caught exception during bulk indexing: java.net.NoRouteToHostException: No route to host (Host unreachable), retrying (attempt #1).`
Also System > Indices -page in UI gives the following errors and do not list any index sets:

Tried also to change unresponding host in elasticsearch_hosts something not even found in DNS and that host was skipped. So problem exists if DNS entry for the host exists but host is not responding to port 9200.
## Possible Solution
Unresponding hosts should be skipped just like the ones with missing DNS entry.
## Steps to Reproduce (for bugs)
1. Set for example two hosts to Graylog’s config variable elasticsearch_hosts and ensure that page System > Indices is listing all the index sets.
2. Shut down either one of the hosts and check the System > Indices page again.
## Context
Situation was noticed when one or our ES coordinating nodes defined in elasticsearch_hosts was down for a maintenance and I tried to open the System > Indices page during that.
## Your Environment
* Graylog Version: 3.2.4
* Elasticsearch Version: 6.8.8
* MongoDB Version: 4.0.17
* Operating System: RHEL 7.7
Contributor guide
Assessment
This issue has not been assessed yet.