hashicorp / hashicorp/nomad

Consul Connect network namespaces incompatible with incremental adoption?

Open
#7,975 2 comments 0 reactions 0 assignees View on GitHub
theme/consul/connect
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

Nomad v0.11.1 (b43457070037800fcc8442c8ff095ff4005dab33)

### Operating system and Environment details

Ubuntu 19.10 on AMD64
Consul v1.7.2
weaveworks/net-plugin:2.6.0

### Issue

We have a Job in Nomad communicating with some services via Consul Connect, and other legacy services via Weave or another Docker network. We expect our tasks to be able to communicate both with Consul Connect'ed services, and services exposed through Weave, but this does not seem to be the case. It does not appear that services can access their upstreams as well as a normal docker network at the same time.

Our minimal HCL is as follows:

```
job "countdash" {
datacenters = ["dc1"]
group "dashboard" {
network {
mode ="bridge"
}
service {
name = "kafka-test"
port = "9002"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "kafka-0"
local_bind_port = 9092
}
}
}
}
}
task "dashboard" {
driver = "docker"
env {
}
config {
image = "ubuntu:latest"
tty = "true"
command = "/bin/bash"
network_mode = "weave"
}
}
}
}
```

This will only function as expected *without* network_mode = "weave".

Without network_mode = "weave":
```
root@356becb95ba8:/# kafkacat -L -b 127.0.0.1:9092
Metadata for all topics (from broker 0: 127.0.0.1:9092/0):
1 brokers:
broker 0 at 127.0.0.1:9092 (controller)
```

With network_mode = "weave", the kafkacat command times out.

We attempted to specify network_mode as taskname-allocation_id, as suggested at https://github.com/hashicorp/nomad/issues/899#issuecomment-433327014, but this does not allow communication outside of the context of a single Linux kernel.

We're looking to adopt Consul Connect but some of our services require a fully connected topology to cluster between them. We'd like to incrementally adopt Consul Connect, but it seems incompatible with our understanding of the network configuration expected while running Connect. Is there an incremental adoption pathway for Connect? Is there a way to allow traffic between the Consul Connect network namespace and existing network namespaces (like Weave's or overlay)?

cc / @dhashs

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the minimal HCL job and reproduce the kafkacat connectivity difference with and without network_mode = "weave" on the stated Nomad, Consul, Ubuntu, and Weave versions. Trace the Connect sidecar and task network-namespace behavior; done means establishing whether incremental Connect adoption with existing overlay networks is supported and documenting or implementing the required behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.