hashicorp / hashicorp/nomad

Fail to update when static ports are busy

Open
#2,067 4 comments 0 reactions 0 assignees View on GitHub
theme/client theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Hi guys,

I just found an abnormal behaviour:

I am running 3 Nomad nodes in client mode, each running a RabbitMQ container with static ports. For my test (and probably for production as well), my ```max_parallel``` (stanza ```update```) is equal to ```count``` (stanza ```group```).

When I update the job file, then apply it, Nomad try to start 3 new containers before stopping old RabbitMQ instances. Unfortunately, because I have only 3 nodes, ports are already busy: new containers cannot start, but old instances are killed anyway.

Could you implement a retry or manage this special case ? It would be pity to have to start 3 more vm, only for rolling-upgrade !

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the rolling update with three Nomad client nodes, RabbitMQ containers, static ports, and max_parallel equal to count. Inspect the update behavior when replacement allocations cannot start because ports are occupied; done should preserve the old instances until replacements can be placed or a retry path is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
rabbitmq
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.