hashicorp / hashicorp/nomad

Non-leader task in taskgroup not being sent interrupt

Open
#6,350 4 comments 0 reactions 0 assignees View on GitHub
stage/needs-verification theme/client type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.9.3 (c5e8b66c3789e4e7f9a83b4e188e9a937eea43ce)

### Operating system and Environment details
Amazon Linux 2

### Issue
Non-leader task in taskgroup receives no interrupt when Nomad decides an allocation should be stopped, say, from a count decrement.

The leader task, which is the task where all the services are registered to Consul, properly handles the `shutdown_delay` and `kill_timeout`, but the secondary task receives no interrupt _until_ the leader task dies. Then it receives an interrupt _because_ the leader task is dead.

This is a problem because the leader task is the sidecar proxy which is the source network for the secondary task, the app itself. Since it receives no interrupts, it is unable to perform graceful shutdown actions until it receives its interrupt, which is when the network is gone since it was attached to the leader task.

### Reproduction steps
1. Create a TaskGroup with two tasks, both using the docker driver
2. Set one task as `leader`
3. Set `"network_mode": "container:-${NOMAD_ALLOC_ID}"` for the secondary
4. Register the healthchecks on the leader
5. Submit the job
6. Reduce the count of the job
7. Verify the leader task gets an interrupt, verify the secondary task gets no interrupt
8. After the `shutdown_delay` and leader terminates, verify the secondary task detects leader is gone
9. Verify the secondary task is sent its interrupt

Leader Task/Proxy:
Screen Shot 2019-09-18 at 11 33 13 PM

Secondary Task/App
Screen Shot 2019-09-18 at 11 33 27 PM

I believe the expected behavior is every task in a taskgroup should receive an interrupt at the same time.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the two-task Docker task group described in the issue, including the leader task, Consul health checks, container network mode, and a reduced job count. Trace the allocation shutdown path for task groups and compare interrupt delivery to the leader and secondary tasks. Done means every task receives its interrupt during shutdown, before the leader's network disappears.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.