No information about placement failure for a port collision when the preemption for the scheduler is activated.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.4.3 (f464aca721d222ae9c1f3df643b3c3aaa20e2da7)
### Operating system and Environment details
Reproduced on RHEL 8 and Fedora 35
### Issue
When there is a placement failure because of a port collision, some information about that placement failure is visible in the status of the job. But when the preemtion for the scheduler is activated, this information is missing:
```
Placement Failure
Task Group "test-port-collision-2":
[...]
```
### Reproduction steps
Run a nomad agent with the following config:
```
name = "test"
region = "global"
datacenter = "dc1"
bind_addr = "0.0.0.0"
data_dir = "/tmp/nomad-data"
client {
enabled = true
}
server {
enabled = true
bootstrap_expect = 1
default_scheduler_config {
preemption_config {
batch_scheduler_enabled = true
system_scheduler_enabled = true
service_scheduler_enabled = true
sysbatch_scheduler_enabled = true
}
}
}
```
And then run a job with a port collision:
```
job "test-port-collision" {
datacenters = ["dc1"]
type = "service"
group "test-port-collision" {
network {
mode = "bridge"
port "myport" {
static = 8000
to = 8000
}
}
task "test-port-collision" {
driver = "exec"
config {
command = "sleep"
args = ["1000"]
}
}
}
group "test-port-collision-2" {
network {
mode = "bridge"
port "otherport" {
static = 8000
to = 8000
}
}
task "test-port-collision-2" {
driver = "exec"
config {
command = "sleep"
args = ["1000"]
}
}
}
}
```
#### Expected Result
In the status of the job, we should see the following information for the placement failure:
```
Placement Failure
Task Group "test-port-collision-2":
* Resources exhausted on 1 nodes
* Dimension "network: reserved port collision otherport=8000" exhausted on 1 nodes
```
#### Actual Result
We have nothing useful in the status of the job:
```
Placement Failure
Task Group "test-port-collision-2":
```
When we remove the preemption part in the config, we get the expected result.
### Nomad logs
What helped us detect the source of the problem is the folowing log line:
```
[DEBUG] worker.service_sched.binpack: preemption not possible : eval_id=f4a3d895-0df1-6f4b-a087-96602c8b5b3d job_id=test-port-collision namespace=default worker_id=8d2ce0ff-03e7-9651-cd7e-a96793b38c34 network_resource="&{bridge 0 [{myport 8000 8000 default}] []}"
```
We have looked for this error in the source code and found it here: https://github.com/hashicorp/nomad/blob/404b47a54c092fddf05a62e938833ed798bd9a53/scheduler/rank.go#L311
In the usual cases, the node is exhausted with the expected error message (line 301). But when there is preemption activated, there is only debug logs but no exhaustion message.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in scheduler/rank.go around the placement and preemption handling referenced at lines 301 and 311. Reproduce the port-collision job with scheduler preemption enabled, then trace why the exhaustion detail is only logged at debug level. Done means the job status reports the reserved-port collision and affected-node count as in the expected result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100