hashicorp / hashicorp/consul

Cannot export `consul` service to peered cluster

Open
#21,624 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

Despite being setup in the `exported-services` `default` configuration entry, `consul` service cannot be exported to a peer.

Content of exported-services

```
{
"Kind": "exported-services",
"Name": "default",
"Services": [
{
"Name": "nomad",
"Consumers": [
{
"Peer": "9c5ff1959e9432b0"
}
]
},
{
"Name": "consul",
"Consumers": [
{
"Peer": "9c5ff1959e9432b0"
}
]
}
],
"CreateIndex": 154,
"ModifyIndex": 896
}
```

image

`nomad` is correctly exported.
`consul` is not

---

#### Reproduction Steps

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 32
services = 24
build:
prerelease =
revision = 9f62fb41
version = 1.19.1
version_metadata =
consul:
acl = enabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 578
max_procs = 2
os = linux
version = go1.22.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 13
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4842
members = 15
query_queue = 0
query_time = 1
```

```
server = false

node_name = "XXX"

encrypt = "XXX"

datacenter = "XXX"
primary_datacenter = "XXX"

retry_join = [
"provider=aws region=eu-west-3 tag_key=XXX tag_value=XXX addr_type=private_v4 service=ec2"
]

node_meta {
ami_id = "XXX"
instance_type = "t3.medium"
instance_purpose = "nomad-client"
availability_zone = "eu-west-3c"
}

ui_config {
enabled = false
}

auto_encrypt {
tls = true
}

tls {
grpc {
verify_incoming = false
}
}

enable_agent_tls_for_checks = true

bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
advertise_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"

client_addr = "0.0.0.0"

data_dir = "/opt/consul"

peering {
enabled = true
}

recursors = [
"169.254.169.253" # AWS DNS
]

ports {
http = 8500 # TCP only
https = 8501 # TCP only
grpc = 8502 # TCP only
grpc_tls = 8503 # TCP only
dns = 8600 # TCP and UDP
server = 8300 # TCP only
serf_lan = 8301 # TCP and UDP
serf_wan = 8302 # TCP and UDP
}

autopilot {
max_trailing_logs = 250
cleanup_dead_servers = true
last_contact_threshold = "200ms"
server_stabilization_time = "10s"
}

telemetry {
disable_hostname = true
prometheus_retention_time = "60s"
}

tls {
defaults {
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "/opt/consul/tls/consul-ca.pem"
key_file = "/opt/consul/tls/consul-key.pem"
cert_file = "/opt/consul/tls/consul-cert.pem"
}

https {
verify_incoming = false
}
}

acl {
enabled = true
default_policy = "deny"
down_policy = "extend-cache"
enable_token_persistence = true
}

connect {
enabled = true
ca_provider = "consul"
}

leave_on_terminate = true
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
```

Server info

```
Error querying agent: Unexpected response code: 403 (Permission denied: anonymous token lacks permission 'agent:read' on "XXX". The anonymous token is used implicitly when a request does not specify a token.)
```

```
server = true

node_name = "XXX"

encrypt = "XXX"

datacenter = "XXX"
primary_datacenter = "XXX"

retry_join = [
"provider=aws region=eu-west-3 tag_key=XXX tag_value=XXX addr_type=private_v4 service=ec2"
]

node_meta {
ami_id = "XXX"
instance_type = "t3a.small"
instance_purpose = "consul"
availability_zone = "eu-west-3c"
}

ui_config {
enabled = true
}

auto_encrypt {
allow_tls = true
}

bootstrap_expect = 3

bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
advertise_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"

client_addr = "0.0.0.0"

data_dir = "/opt/consul"

peering {
enabled = true
}

recursors = [
"169.254.169.253" # AWS DNS
]

ports {
http = 8500 # TCP only
https = 8501 # TCP only
grpc = 8502 # TCP only
grpc_tls = 8503 # TCP only
dns = 8600 # TCP and UDP
server = 8300 # TCP only
serf_lan = 8301 # TCP and UDP
serf_wan = 8302 # TCP and UDP
}

autopilot {
max_trailing_logs = 250
cleanup_dead_servers = true
last_contact_threshold = "200ms"
server_stabilization_time = "10s"
}

telemetry {
disable_hostname = true
prometheus_retention_time = "60s"
}

tls {
defaults {
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "/opt/consul/tls/consul-ca.pem"
key_file = "/opt/consul/tls/consul-key.pem"
cert_file = "/opt/consul/tls/consul-cert.pem"
}

https {
verify_incoming = false
}
}

acl {
enabled = true
default_policy = "deny"
down_policy = "extend-cache"
enable_token_persistence = true
}

connect {
enabled = true
ca_provider = "consul"
}

leave_on_terminate = true
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
```

### Operating system and Environment details

Consul version: 1.19.1
OS: Linux
Architecture: amd64/arm64
Distribution: Amazon Linux 2023

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the peered-cluster setup with the exported-services entry and the provided Consul 1.19.1 configuration; done means the consul service is exported to the peer as nomad is.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.