hashicorp / hashicorp/nomad

Not showing warning when using a taken GPU

Open
#18,364 3 comments 0 reactions 0 assignees View on GitHub
hcc/jira stage/accepted theme/devices theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
nomad server: `Nomad v1.6.1` + patch with #18141
nomad client: `Nomad v1.6.1` + patch with #18141

### Operating system and Environment details
Plugin "nomad-driver-podman" v0.5.1
Plugin "nomad-device-nvidia" v1.0.0

### Issue
I created a patch with the solution provides in #18141 to test on our cluster. While testing I discovered that if a GPU or multiple GPU are already used in jobs, nomad will not give a warning about this and will place the job without using those GPUs.

### Reproduction steps
1. Create a job file where you use multiple GPUs and set constraint based on their UUID.
2. Create a new job file where you use one or more GPUs that is already used in the first job.

#### Expected Result
Throw an warning like `WARNING: Failed to place all allocations.`

#### Actual Result
Place the job on the client.

### Job file (if appropriate)
Job file 1:
```
job "test-2070-2" {
datacenters = ["dc1"]
group "test-2070-2" {

restart {
attempts=0
}
count=1
task "test-2070-2" {
driver = "podman"
config {
image = "image_with_gpu"
}
resources {
cpu = 2650
memory = 8192
device "nvidia/gpu" {
count = 2

constraint {
attribute = "${device.model}"
value = "NVIDIA GeForce RTX 2070 SUPER"
}

constraint {
attribute = "${device.ids}"
operator = "set_contains"
value = "GPU-9b5df054-6f08-f35c-9c4c-5709b19efea5,GPU-1846fc5f-8c71-bfab-00e1-9c190dd88ed7"
}

}
}
}
}
}
```

Job file 2:
```
job "test-2070-2" {
datacenters = ["dc1"]
group "test-2070-2" {

restart {
attempts=0
}
count=1
task "test-2070-2" {
driver = "podman"
config {
image = "image_with_gpu"
}
resources {
cpu = 2650
memory = 8192
device "nvidia/gpu" {
count = 1

constraint {
attribute = "${device.model}"
value = "NVIDIA GeForce RTX 2070 SUPER"
}

constraint {
attribute = "${device.ids}"
operator = "set_contains"
value = "GPU-9b5df054-6f08-f35c-9c4c-5709b19efea5"
}

}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the behavior with the two GPU job files and the stated Nomad, nomad-driver-podman, and nomad-device-nvidia versions. Start by tracing device allocation and placement handling for UUID constraints, then identify the relevant scheduler or device-allocation tests. Done means a job requesting an already allocated GPU is rejected or reports the expected placement warning without disrupting valid allocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.