Graylog2 / Graylog2/graylog2-server
Datanode - Datanodes running under docker bridge networks can not join clusters across hosts
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
All datanodes should be able to join a cluster regardless with any routeable docker network configuration.
## Current Behavior
OpenSearch publishes the internal docker network address of the node which is non-routeable to other cluster members. This is because docker creates a host file on all containers which maps the hostname to it's internal network address, it also runs an internal DNS server which contains the same information. Datanode sets `network.publish_host` to be the hostname of the container. OpenSearch when it attempts to resolve `network.publish_host` gets the local address from the host file (or docker internal DNS server) and adds it to the publish list.
Example:
```
2024-11-15T16:44:14.489Z INFO [OpensearchProcessImpl] [2024-11-15T16:44:14,489][INFO ][o.o.t.TransportService ] [datanode2.gltest.internal] publish_address {datanode2.gltest.internal/172.18.0.2:9300}, bound_addresses {[::]:9300}
```
The address `172.18.0.2` is on an internal docker bridge network. Externally it resolves to a routeable address.
Host file:
```
$ docker exec -it datanode-datanode-1 cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.2 datanode2.gltest.internal datanode2
```
External resolution:
```
$ host datanode2.gltest.internal
datanode2.gltest.internal has address 10.100.100.93
```
Generated configuration (only the networky bits):
```
$ docker exec -it datanode-datanode-1 cat /var/lib/graylog-datanode/opensearch/config/opense
arch/opensearch.yml
---
network.host: "0.0.0.0"
http.port: "9200"
transport.port: "9300"
cluster.name: "datanode-cluster"
node.name: "datanode2.gltest.internal"
cluster.initial_cluster_manager_nodes: "datanode2.gltest.internal,datanode1.gltest.internal"
network.bind_host: "0.0.0.0"
network.publish_host: "datanode2.gltest.internal"
```
## Possible Solution
Allow setting `network.publish_host` independently of the host name. This occurs when the hostname matches `network.publish_host` in a docker bridge network.
Example:
Hostname: `datanode.example.org` - Publish_Host: `datanode.example.org` - Local Docker IP is published by OpenSearch
Hostname: `datanode` - Publish_Host: `datanode.example.org` - IP resolved normally for `datanode.example.org` and would be published by OpenSearch.
I can't test changing `network.publish_host` since datanode sets it automatically but I can show the DNS resolution changes.
Hostname: `datanode2`
```
$ docker exec -it datanode-datanode-1 getent hosts datanode2
172.18.0.2 datanode2
$ docker exec -it datanode-datanode-1 getent hosts datanode2.gltest.internal
10.100.100.93 datanode2.gltest.internal
```
## Steps to Reproduce (for bugs)
You need two hosts that can communicate across the network.
Host 1 - Graylog, MongoDB and Datanode
Host 2 - Docker running datanode.
DNS or `/etc/hosts` on all hosts to resolve the intended nodes (in my case hosts file and datanode1.gltest.internal and datanode2.gltest.internal)
Example compose config:
```
services:
datanode:
image: "graylog/graylog-datanode:6.1"
hostname: "datanode2.gltest.internal"
environment:
GRAYLOG_DATANODE_NODE_NAME: "datanode2.gltest.internal"
GRAYLOG_DATANODE_NODE_ID_FILE: "/var/lib/graylog-datanode/node-id"
GRAYLOG_DATANODE_PASSWORD_SECRET: "somepasswordpepper"
GRAYLOG_DATANODE_ROOT_PASSWORD_SHA2: "_your_root_password_in_sha256"
GRAYLOG_DATANODE_MONGODB_URI: "mongodb://mongodbhost:27017/graylog"
GRAYLOG_DATANODE_OPENSEARCH_DATA_LOCATION: "/var/lib/opensearch"
ulimits:
memlock:
hard: -1
soft: -1
nofile:
soft: 65536
hard: 65536
ports:
- "8999:8999/tcp" # DataNode API
- "9200:9200/tcp"
- "9300:9300/tcp"
networks:
- datanode
volumes:
- "./datanode_config:/var/lib/graylog-datanode"
- "./opensearch_data:/var/lib/opensearch"
restart: "on-failure"
networks:
datanode:
driver: "bridge"
```
1. Start Host 1 and all services, make sure that datanode initializes and Graylog is accessible.
2. Start Host 2 and bring up docker datanode.
3. Cluster will never form, errors will occurs as the datanodes will attempt to connect to the docker internal non-routable IP.
```
2024-11-15T17:07:07.929Z INFO [OpensearchProcessImpl] [2024-11-15T17:07:07,926][WARN ][o.o.d.HandshakingTransportAddressConnector] [datanode1.gltest.internal] [connectToRemoteMasterNode[10.100.100.93:9300]] completed handshake with [{datanode2.gltest.internal}{qzR5I81ZSiiNCC5FM8fB7Q}{SktF0u99QeiyhVDFcvOq7A}{datanode2.gltest.internal}{172.18.0.2:9300}{dimrs}{shard_indexing_pressure_enabled=true}] but followup connection failed
2024-11-15T17:07:07.930Z INFO [OpensearchProcessImpl] org.opensearch.transport.ConnectTransportException: [datanode2.gltest.internal][172.18.0.2:9300] connect_exception
2024-11-15T17:07:07.932Z INFO [OpensearchProcessImpl] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: datanode2.gltest.internal/172.18.0.2:9300
```
## Context
Docker bridges provide service isolation from the wider network, this is the default configuration when you do not specify a network type in docker compose. I would image the same issue could happen in Kubernetes or behind other load balancers, proxies, NAT, etc.
## Your Environment
Two brand new for testing only Ubuntu 22.04 servers running on Hyper-V.
* Graylog Version: 6.1.2 (docker)
* Java Version: Embedded
* Datanode Version: 6.1.2 (docker)
* MongoDB Version: 7.0.15 (docker)
* Operating System: Ubuntu 22.04
* Browser version: Firefox 132.0.2
Contributor guide
Assessment
This issue has not been assessed yet.