Nomad registered service check fails without any output.
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
Done nomad / consul integration as per https://developer.hashicorp.com/nomad/docs/integrations/consul-integration
Seems like service is registered, but the health check is not working (for both nomad-server and client)
There also seems to be a stale check which is not being deregistered (This is a single server cluster - so it is not from another server)
Surprisingly, I cannot see any `[Error] http check failed` kinda logs - only the service registration logs.
---
#### Reproduction Steps
Consul ACL for nomad
```
agent_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
service_prefix "" {
policy = "write"
}
acl = "write" # Only for server not for client
```
Use CONSUL_HTTP_TOKEN env var in nomad and enable consul integration.
Tried changing advertising address / etc. in nomad, could not get it working.
### Consul info for both Client and Server
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 192df66a
version = 1.16.0
version_metadata =
consul:
acl = enabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.1.1.1:8300
server = true
raft:
applied_index = 681355
commit_index = 681355
fsm_pending = 0
last_contact = 0
last_log_index = 681355
last_log_term = 218
last_snapshot_index = 671815
last_snapshot_term = 207
latest_configuration = [{Suffrage:Voter ID:31172a93-71b5-9e17-83b5-7bc8e550e51c Address:10.1.1.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 218
runtime:
arch = arm64
cpu_count = 4
goroutines = 183
max_procs = 4
os = linux
version = go1.20.4
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 218
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 530
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Consul 1.16 / Ubuntu 22.04
### Log Fragments
Contributor guide
Research direction
Start by reproducing the Nomad/Consul integration on Consul 1.16 and Ubuntu 22.04 with the stated ACL policy and CONSUL_HTTP_TOKEN configuration. Compare service registration, health-check results, stale-check deregistration, and agent logs; done means checks work for both Nomad server and client, stale checks are removed, and failures provide diagnostic output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100