hashicorp / hashicorp/nomad

Consul Connect with Consul ACL enabled in a federated cluster Nomad Task doesn't get registered

Open
#8,019 19 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/consul theme/consul/connect theme/service-discovery/consul type/question
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Output from `nomad version`
```bash
root@nyc-consul-server:/vagrant/provision/terraform/tf_cluster/secondary# nomad version
Nomad v0.11.2 (807cfebe90d56f9e5beec3e72936ebe86acc8ce3)
root@nyc-consul-server:/vagrant/provision/terraform/tf_cluster/secondary# consul version
Consul v1.7.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```
### Operating system and Environment details
Linux / ubuntu

### Issue

Have a consul federated cluster with TLS and ACLs enabled

deploy a consul connect job // count dash can work

deploy one service in dc1 and the second service in dc2

dc1 can deploy everything as expected

dc2 needs a replication acl token, and needs consul dc1 token in order to work

and from here everything in dc2 ask for a token, nomad doesn't get / read the consul token environment variable that is set in the host any more and it doesn't register the services correctly

### Job file (if appropriate)
```hcl
job "cinemas" {

datacenters = ["nyc-ncv"]
region = "nyc-region"
type = "service"

group "payment-api" {
count = 1

network {
mode = "bridge"

port "healthcheck" {
to = -1
}
}

service {
name = "payment-api"
port = "3000"

check {
name = "payment-api-health"
port = "healthcheck"
type = "http"
protocol = "http"
path = "/ping"
interval = "5s"
timeout = "2s"
expose = true
}

connect {
sidecar_service {}
}
}

task "payment-api" {
driver = "docker"

config {
image = "crizstian/payment-service-go:v0.4"
}

env {
DB_SERVERS = "mongodb1.query.consul:27017,mongodb2.query.consul:27018,mongodb3.query.consul:27019"
SERVICE_PORT = "3000"
CONSUL_IP = "consul.service.consul"
CONSUL_SCHEME = "https"
CONSUL_HTTP_SSL = "true"
}

resources {
cpu = 50
memory = 50
}
}
}
}
```
### Nomad Client logs (if appropriate)
```bash

root@nyc-consul-server:/vagrant/provision/terraform/tf_cluster/secondary# nomad status cinemas
ID = cinemas
Name = cinemas
Submit Date = 2020-05-19T19:26:35Z
Type = service
Priority = 50
Datacenters = nyc-ncv
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost
payment-api 0 0 1 0 0 0

Latest Deployment
ID = 97bc5bdc
Status = running
Description = Deployment is running

Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
payment-api 1 1 0 0 2020-05-19T19:36:35Z

Allocations
ID Node ID Task Group Version Desired Status Created Modified
788f0165 5fd6be7a payment-api 0 run running 12s ago 9s ago
root@nyc-consul-server:/vagrant/provision/terraform/tf_cluster/secondary# nomad logs 788f0165
Allocation "788f0165" is running the following tasks:
* payment-api
* connect-proxy-payment-api

Please specify the task.
root@nyc-consul-server:/vagrant/provision/terraform/tf_cluster/secondary# nomad logs 788f0165 connect-proxy-payment-api
root@nyc-consul-server:/vagrant/provision/terraform/tf_cluster/secondary# nomad logs -stderr 788f0165 connect-proxy-payment-api
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:238] initializing epoch 0 (hot restart version=disabled)
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:240] statically linked extensions:
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:242] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:245] filters.http: envoy.buffer,envoy.cors,envoy.csrf,envoy.ext_authz,envoy.fault,envoy.filters.http.dynamic_forward_proxy,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.original_src,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:248] filters.listener: envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:251] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.filters.network.zookeeper_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:253] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:255] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.tracers.opencensus,envoy.zipkin
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:258] transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:261] transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
[2020-05-19 19:26:38.228][1][info][main] [source/server/server.cc:267] buffer implementation: old (libevent)
[2020-05-19 19:26:38.233][1][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-05-19 19:26:38.236][1][info][main] [source/server/server.cc:322] admin address: 127.0.0.1:19001
[2020-05-19 19:26:38.236][1][info][main] [source/server/server.cc:432] runtime: layers:
- name: static_layer
static_layer:
envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1: true
envoy.deprecated_features:envoy.api.v2.Cluster.tls_context: true
envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name: true
[2020-05-19 19:26:38.237][1][warning][runtime] [source/common/runtime/runtime_impl.cc:497] Skipping unsupported runtime layer: name: "static_layer"
static_layer {
fields {
key: "envoy.deprecated_features:envoy.api.v2.Cluster.tls_context"
value {
bool_value: true
}
}
fields {
key: "envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name"
value {
bool_value: true
}
}
fields {
key: "envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1"
value {
bool_value: true
}
}
}

[2020-05-19 19:26:38.237][1][info][config] [source/server/configuration_impl.cc:61] loading 0 static secret(s)
[2020-05-19 19:26:38.237][1][info][config] [source/server/configuration_impl.cc:67] loading 1 cluster(s)
[2020-05-19 19:26:38.244][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:144] cm init: initializing cds
[2020-05-19 19:26:38.245][1][info][config] [source/server/configuration_impl.cc:71] loading 0 listener(s)
[2020-05-19 19:26:38.246][1][info][config] [source/server/configuration_impl.cc:96] loading tracing configuration
[2020-05-19 19:26:38.246][1][info][config] [source/server/configuration_impl.cc:116] loading stats sink configuration
[2020-05-19 19:26:38.246][1][info][main] [source/server/server.cc:516] starting main dispatch loop
[2020-05-19 19:26:38.252][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 7, permission denied
[2020-05-19 19:26:38.252][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:148] cm init: all clusters initialized
[2020-05-19 19:26:38.252][1][info][main] [source/server/server.cc:500] all clusters initialized. initializing init manager
[2020-05-19 19:26:38.638][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 7, permission denied
[2020-05-19 19:26:38.638][1][info][config] [source/server/listener_manager_impl.cc:761] all dependencies initialized. starting workers
[2020-05-19 19:26:39.002][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 7, permission denied
[2020-05-19 19:26:41.349][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 7, permission denied
[2020-05-19 19:26:43.117][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 7, permission denied
[2020-05-19 19:26:57.793][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 7, permission denied
```

Captura de Pantalla 2020-05-19 a la(s) 14 34 41

Even the consul health check defined also doesn't work

Captura de Pantalla 2020-05-19 a la(s) 14 38 16

i also tried setting the consul token at the job level and as well a tried to set the token like the following:

```hcl
connect {
sidecar_service {
proxy {
config {
token = "........"
}
}
}
}
```

Looks like if consul federation works properly but looks like the acl system isn't working properly or something is missing in my acl config for a federation cluster

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.