Not showing warning when using a taken GPU
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
nomad server: `Nomad v1.6.1` + patch with #18141
nomad client: `Nomad v1.6.1` + patch with #18141
### Operating system and Environment details
Plugin "nomad-driver-podman" v0.5.1
Plugin "nomad-device-nvidia" v1.0.0
### Issue
I created a patch with the solution provides in #18141 to test on our cluster. While testing I discovered that if a GPU or multiple GPU are already used in jobs, nomad will not give a warning about this and will place the job without using those GPUs.
### Reproduction steps
1. Create a job file where you use multiple GPUs and set constraint based on their UUID.
2. Create a new job file where you use one or more GPUs that is already used in the first job.
#### Expected Result
Throw an warning like `WARNING: Failed to place all allocations.`
#### Actual Result
Place the job on the client.
### Job file (if appropriate)
Job file 1:
```
job "test-2070-2" {
datacenters = ["dc1"]
group "test-2070-2" {
restart {
attempts=0
}
count=1
task "test-2070-2" {
driver = "podman"
config {
image = "image_with_gpu"
}
resources {
cpu = 2650
memory = 8192
device "nvidia/gpu" {
count = 2
constraint {
attribute = "${device.model}"
value = "NVIDIA GeForce RTX 2070 SUPER"
}
constraint {
attribute = "${device.ids}"
operator = "set_contains"
value = "GPU-9b5df054-6f08-f35c-9c4c-5709b19efea5,GPU-1846fc5f-8c71-bfab-00e1-9c190dd88ed7"
}
}
}
}
}
}
```
Job file 2:
```
job "test-2070-2" {
datacenters = ["dc1"]
group "test-2070-2" {
restart {
attempts=0
}
count=1
task "test-2070-2" {
driver = "podman"
config {
image = "image_with_gpu"
}
resources {
cpu = 2650
memory = 8192
device "nvidia/gpu" {
count = 1
constraint {
attribute = "${device.model}"
value = "NVIDIA GeForce RTX 2070 SUPER"
}
constraint {
attribute = "${device.ids}"
operator = "set_contains"
value = "GPU-9b5df054-6f08-f35c-9c4c-5709b19efea5"
}
}
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the behavior with the two GPU job files and the stated Nomad, nomad-driver-podman, and nomad-device-nvidia versions. Start by tracing device allocation and placement handling for UUID constraints, then identify the relevant scheduler or device-allocation tests. Done means a job requesting an already allocated GPU is rejected or reports the expected placement warning without disrupting valid allocations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100