hashicorp / hashicorp/nomad

CNI plugin hooks occasionally fail due to duplicate address allocation with system jobs

Open
#11,272 3 comments 4 reactions 0 assignees View on GitHub
stage/needs-investigation theme/networking type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

```
$ nomad version
Nomad v1.1.3 (5e94e4c06cdc18916e9c0d5f0e8c2130d6cfd2ac)
```

### Operating system and Environment details
Void Linux x86_64-musl, CNI plugins 1.0.1, Docker 20.10.6

### Issue

Occasionally Nomad will try to assign the same IP to a task that's already been allocated. This feels like state is not being correctly reconciled when Nomad starts up on a node that rebooted.

### Reproduction steps

Launch some system tasks, reboot, nomad has some chance of not bringing back system tasks due to IP collisions.

#### Expected Result

Nomad should be able to survive a reboot and restore system services to a host.

#### Actual Result

Jobs fail to restore because duplicate IP addressing is not allowed.

### Nomad Server logs (if appropriate)
Sorry, logging is a system service and unavailable during this issue :/

### Nomad Client logs (if appropriate)
N/A

### Screenshot of a dead task:
![image](https://user-images.githubusercontent.com/2321814/136148622-3ae9b9bc-8aeb-486a-b950-0793f1a9c521.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at Nomad startup on a rebooted client and trace system-task restoration through the CNI plugin hooks; the report identifies duplicate IP allocation as the failure point. Reproduce with system tasks and inspect the available client or CNI evidence; done means system services restore without duplicate addresses.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.