hashicorp / hashicorp/nomad

No information about placement failure for a port collision when the preemption for the scheduler is activated.

Open
#15,975 0 comments 2 reactions 0 assignees View on GitHub
theme/preemption theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.4.3 (f464aca721d222ae9c1f3df643b3c3aaa20e2da7)

### Operating system and Environment details
Reproduced on RHEL 8 and Fedora 35

### Issue
When there is a placement failure because of a port collision, some information about that placement failure is visible in the status of the job. But when the preemtion for the scheduler is activated, this information is missing:
```
Placement Failure
Task Group "test-port-collision-2":

[...]
```

### Reproduction steps
Run a nomad agent with the following config:
```
name = "test"
region = "global"
datacenter = "dc1"
bind_addr = "0.0.0.0"
data_dir = "/tmp/nomad-data"

client {
enabled = true
}

server {
enabled = true

bootstrap_expect = 1

default_scheduler_config {
preemption_config {
batch_scheduler_enabled = true
system_scheduler_enabled = true
service_scheduler_enabled = true
sysbatch_scheduler_enabled = true
}
}
}
```

And then run a job with a port collision:
```
job "test-port-collision" {
datacenters = ["dc1"]
type = "service"

group "test-port-collision" {
network {
mode = "bridge"
port "myport" {
static = 8000
to = 8000
}
}

task "test-port-collision" {
driver = "exec"

config {
command = "sleep"

args = ["1000"]
}
}
}

group "test-port-collision-2" {
network {
mode = "bridge"
port "otherport" {
static = 8000
to = 8000
}
}

task "test-port-collision-2" {
driver = "exec"

config {
command = "sleep"

args = ["1000"]
}
}
}
}

```

#### Expected Result
In the status of the job, we should see the following information for the placement failure:
```
Placement Failure
Task Group "test-port-collision-2":
* Resources exhausted on 1 nodes
* Dimension "network: reserved port collision otherport=8000" exhausted on 1 nodes

```

#### Actual Result
We have nothing useful in the status of the job:
```
Placement Failure
Task Group "test-port-collision-2":

```

When we remove the preemption part in the config, we get the expected result.

### Nomad logs
What helped us detect the source of the problem is the folowing log line:
```
[DEBUG] worker.service_sched.binpack: preemption not possible : eval_id=f4a3d895-0df1-6f4b-a087-96602c8b5b3d job_id=test-port-collision namespace=default worker_id=8d2ce0ff-03e7-9651-cd7e-a96793b38c34 network_resource="&{bridge 0 [{myport 8000 8000 default}] []}"
```

We have looked for this error in the source code and found it here: https://github.com/hashicorp/nomad/blob/404b47a54c092fddf05a62e938833ed798bd9a53/scheduler/rank.go#L311

In the usual cases, the node is exhausted with the expected error message (line 301). But when there is preemption activated, there is only debug logs but no exhaustion message.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in scheduler/rank.go around the placement and preemption handling referenced at lines 301 and 311. Reproduce the port-collision job with scheduler preemption enabled, then trace why the exhaustion detail is only logged at debug level. Done means the job status reports the reserved-port collision and affected-node count as in the expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.