hashicorp / hashicorp/nomad

[feature] Agent configuration to define action on loss of connectivity

Open
#25,936 1 comment 0 reactions 0 assignees View on GitHub
hcc/jira stage/needs-discussion theme/client theme/core type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

# Note: this is a re-file of the #24679 since that got sidetracked into debugging an issue with options defined in the job spec. Please see comments there for more context.

### Proposal

Currently when the loss of connectivity between an agent(s) and servers occurs the servers will attempt to reschedule the job according to the job configuration. However, there is no way to configure what an agent should do in such situation. As it currently stands, the agent will continue running the pre-existing allocations indefinitely, which in some cases may be undesirable. While some cases can be handled through application logic, this assumes that 1 - network split would have an impact on applications logic (i.e. loss of connectivity to some dependency at the same time), 2 - the application has been implemented with such case in mind. This leaves 2 groups of cases uncovered -- when connectivity between nomad agents and servers would have no impact on application functionality (i.e. all dependencies are still reachable), and when the application is a legacy one that does not necessarily check and handle this case.

Moreover on the orphaned agents, the allocations that were manually killed will be restarted by the agent (as tested in Nomad 1.8.1)

The proposed change is a high-level description of what may be considered as an addition.

To allow for better handling of such cases I think it would be beneficial for the agent to be configured such as that it would shutdown tasks/allocations running on it, if it becomes "orphaned" with a configured interval to wait. For example having 2 additional options in the agent section would allow for better handling of above described cases:

```
shutdown_orphaned_tasks = # enable/disable shutting down local tasks when the agent becomes orphaned. ideally the shutdown would use configured kill_signal of a task. defaulting to false will mean no breaking changes

shutdown_orphaned_tasks_timeout = # set a timeout that starts upon agent becoming disconnected from servers
```

### Use-cases
1. Network separation between agents and servers when the applications should not be allowed to reschedule while another instance is running

### Attempted Solutions
There is generally loss of control of the agent happens with the network partitions, ~currently the only way to shutdown processes running on the agent is to manually kill the process~ there is currently no way to simply stop application on the orphaned node as the agent will restart the application if it is manually killed (according to job definition), which is far from ideal, in our environment we would have to implement some watchdog process to run on every agent node to monitor whether it is connected to servers and then continuously kill all running tasks by itself (there isn't even an API on agent to list locally running tasks and stop them, which means whatever watchdog we will use will have to do it via other means).

**There is currently a [disconnect.stop_after](https://developer.hashicorp.com/nomad/docs/job-specification/disconnect#stop_after) option of the job spec. However, in an environment where job specs are maintained by developers but the guarantees of the system are placed on the infra team this is not sufficient in my opinion. We wouldn't be able to read through every change of the job specs before it is rolled out and if we did it would become a bottleneck in the overall process.**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the referenced issue #24679 and the existing disconnect.stop_after job-specification option. Determine how agent/server connectivity loss and orphaned allocations are handled today, then define the agent configuration behavior and its interaction with task kill settings. Done means the proposed options and orphaned-task handling are implemented, documented, and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.