hashicorp / hashicorp/consul

Nomad registered service check fails without any output.

Open
#18,071 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
Done nomad / consul integration as per https://developer.hashicorp.com/nomad/docs/integrations/consul-integration
Seems like service is registered, but the health check is not working (for both nomad-server and client)

Screenshot 2023-07-11 at 5 33 27 PM

There also seems to be a stale check which is not being deregistered (This is a single server cluster - so it is not from another server)
image

Surprisingly, I cannot see any `[Error] http check failed` kinda logs - only the service registration logs.

---

#### Reproduction Steps

Consul ACL for nomad
```
agent_prefix "" {
policy = "read"
}

node_prefix "" {
policy = "read"
}

service_prefix "" {
policy = "write"
}

acl = "write" # Only for server not for client
```

Use CONSUL_HTTP_TOKEN env var in nomad and enable consul integration.

Tried changing advertising address / etc. in nomad, could not get it working.

### Consul info for both Client and Server

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 192df66a
version = 1.16.0
version_metadata =
consul:
acl = enabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.1.1.1:8300
server = true
raft:
applied_index = 681355
commit_index = 681355
fsm_pending = 0
last_contact = 0
last_log_index = 681355
last_log_term = 218
last_snapshot_index = 671815
last_snapshot_term = 207
latest_configuration = [{Suffrage:Voter ID:31172a93-71b5-9e17-83b5-7bc8e550e51c Address:10.1.1.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 218
runtime:
arch = arm64
cpu_count = 4
goroutines = 183
max_procs = 4
os = linux
version = go1.20.4
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 218
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 530
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Consul 1.16 / Ubuntu 22.04

### Log Fragments

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Nomad/Consul integration on Consul 1.16 and Ubuntu 22.04 with the stated ACL policy and CONSUL_HTTP_TOKEN configuration. Compare service registration, health-check results, stale-check deregistration, and agent logs; done means checks work for both Nomad server and client, stale checks are removed, and failures provide diagnostic output.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.