Nomad port collision issue with alloc from different groups
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.8.4 (Build: 22ab32e6, Date: 2024-09-17T20:18:34Z)
### Environment
Debian GNU/Linux 12
### Issue
Port collision occurring during deployment when Nomad allocates the same port (30943) to a new allocation while the previous allocation is still shutting down.
### Reproduction steps
1. Initial allocation (692e5352) using port 30943
2. New allocation (dc4c7ccc) assigned same port 30943 (allocations belong to same job but to different groups)
3. Timeline:
- 09:21:19 - Old allocation receives stop signal
- 09:21:27 - New allocation attempts to start
- 09:21:29 - New allocation fails with port collision
- 09:21:38 - Old allocation finally terminates
#### Degailed logs
nomad logs:
```
2025-01-20T09:21:19.057Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type=Killing msg="Sent interrupt. Waiting 3m0s before force killing" failed=false
2025-01-20T09:21:19.137Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type="Waiting for shutdown delay" msg="Waiting for shutdown_delay of 5s before killing the task." failed=false
2025-01-20T09:21:19.159Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=dc4c7ccc-156e-b889-8e77-3910d223b48a task=java type=Received msg="Task received by client" failed=false
2025-01-20T09:21:27.171Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=dc4c7ccc-156e-b889-8e77-3910d223b48a task=java type="Task Setup" msg="Building Task Directory" failed=false
2025-01-20T09:21:27.575Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=dc4c7ccc-156e-b889-8e77-3910d223b48a task=java type=Started msg="Task started by client" failed=false
2025-01-20T09:21:38.103Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type=Terminated msg="Exit Code: 143, Exit Message: \"Docker container exited with non-zero exit code: 143\"" failed=false
2025-01-20T09:21:38.126Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type=Killed msg="Task successfully killed" failed=false
2025-01-20T09:21:38.138Z [INFO] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java plugin=/usr/bin/nomad id=642473
2025-01-20T09:21:38.138Z [INFO] client.gc: marking allocation for GC: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22
```
new alloc app logs:
```
Jan 20 09:21:28 15f9cf722360[2894512]: + exec java ... port=30943
Jan 20 09:21:29 15f9cf722360[2894512]: Could not start: java.net.BindException: Address already in use
```
old alloc app logs:
```
Jan 20 07:03:15 ac3bf2eca298[2894512]: + exec java ... port=30943
Jan 20 09:21:24 ac3bf2eca298[2894512]: Shutdown called
```
### Expected Result
Nomad should prevent port collisions by ensuring previous allocation fully releases the port before allowing reuse.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Nomad's allocation shutdown and port reuse lifecycle using the reported allocation IDs and port 30943; no source file or test is named in the issue. Reproduce the timing between the stop signal and old allocation termination, then verify that a replacement allocation cannot bind the port until the previous allocation has fully released it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, java
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100