hashicorp / hashicorp/nomad

Nomad uniqueness of consul agent is confusing

Open
#7,639 6 comments 0 reactions 0 assignees View on GitHub
theme/consul theme/service-discovery/consul type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad agents are expected to have unique mapping to consul agents: No two nomad agents should communicate to the same consul agent.

Though documented[1], the failure mode is very confusing and is hard to debug. It appears as nomad job services get removed unexpectedly and for no obvious reason, and without any log messages.

We ought to improve the situation. Consider the following:

* Alert operators better when condition is not met, e.g
* Nomad agent report an error or a warning on launch if two nomad agents are interacting with the same agent
* Use consul's lock/sessions to ensure this properly

* Remove that restriction and allow multiple nomad agents to connect to the same consul
* When removing querying consul, use labels/tags to filter by nomad agent

[1] https://github.com/hashicorp/nomad/blob/v0.11.0-beta2/website/pages/docs/configuration/consul.mdx#consul-stanza .

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked Consul stanza documentation and the issue's description of Nomad–Consul agent uniqueness. Determine which of the proposed directions is intended—operator warnings or sessions, or removing the restriction with agent filtering—and define observable behavior before locating the relevant integration code and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.