hashicorp / hashicorp/nomad

Consul Connect for sharded tasks

Open
#11,032 3 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/consul/connect type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
Allow for connecting multiple instances of the same task (eg. when job.group.count > 1) to each other.

### Example usage
```hcl
job "rabbitmq" {
...
group "rabbitmq" {
count = 3
network {
mode = "bridge"
port "rabbit" {}
}
service {
name = "rabbitmq"
port = "rabbit"
connect {
sidecar_service {
proxy {
upstreams {
self = true
local_bind_port = "5672"
}
}
}
}
}
}
task "rabbitmq" {
template {
data = <

### Use-cases
Clustered tasks such as the mentioned rabbitmq or other message brokers, databases/search engines such as elasticsearch and other HA services that need to communicate with each other.

### Attempted Solutions
- I have unsuccessfully tried using host network (on one nomad node, so running with count > 1 fails due to port collision) to gain access to Consul instead (rabbitmq has consul plugin)
- I've attempted to ask if there's something I'm missing here: https://discuss.hashicorp.com/t/rabbitmq-with-consul-connect-using-nomad/28018
and here: https://gitter.im/hashicorp-nomad/Lobby?at=6111f5f07bc44d0a47136194 unsuccessfully
- I've attempted various stuff that I don't even remember anymore :(

I don't know if this is something that has to be done on Nomad side or Consul side.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by determining whether sharded self-upstreams belong in Nomad’s Consul Connect integration or Consul, then define the behavior for NOMAD_UPSTREAM_SELF and validate it against the RabbitMQ-style clustered-task example.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.