Consul Connect network namespaces incompatible with incremental adoption?
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.11.1 (b43457070037800fcc8442c8ff095ff4005dab33)
### Operating system and Environment details
Ubuntu 19.10 on AMD64
Consul v1.7.2
weaveworks/net-plugin:2.6.0
### Issue
We have a Job in Nomad communicating with some services via Consul Connect, and other legacy services via Weave or another Docker network. We expect our tasks to be able to communicate both with Consul Connect'ed services, and services exposed through Weave, but this does not seem to be the case. It does not appear that services can access their upstreams as well as a normal docker network at the same time.
Our minimal HCL is as follows:
```
job "countdash" {
datacenters = ["dc1"]
group "dashboard" {
network {
mode ="bridge"
}
service {
name = "kafka-test"
port = "9002"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "kafka-0"
local_bind_port = 9092
}
}
}
}
}
task "dashboard" {
driver = "docker"
env {
}
config {
image = "ubuntu:latest"
tty = "true"
command = "/bin/bash"
network_mode = "weave"
}
}
}
}
```
This will only function as expected *without* network_mode = "weave".
Without network_mode = "weave":
```
root@356becb95ba8:/# kafkacat -L -b 127.0.0.1:9092
Metadata for all topics (from broker 0: 127.0.0.1:9092/0):
1 brokers:
broker 0 at 127.0.0.1:9092 (controller)
```
With network_mode = "weave", the kafkacat command times out.
We attempted to specify network_mode as taskname-allocation_id, as suggested at https://github.com/hashicorp/nomad/issues/899#issuecomment-433327014, but this does not allow communication outside of the context of a single Linux kernel.
We're looking to adopt Consul Connect but some of our services require a fully connected topology to cluster between them. We'd like to incrementally adopt Consul Connect, but it seems incompatible with our understanding of the network configuration expected while running Connect. Is there an incremental adoption pathway for Connect? Is there a way to allow traffic between the Consul Connect network namespace and existing network namespaces (like Weave's or overlay)?
cc / @dhashs
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the minimal HCL job and reproduce the kafkacat connectivity difference with and without network_mode = "weave" on the stated Nomad, Consul, Ubuntu, and Weave versions. Trace the Connect sidecar and task network-namespace behavior; done means establishing whether incremental Connect adoption with existing overlay networks is supported and documenting or implementing the required behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- infrastructure, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100