Nomad fails to clean up reserved cores
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
```
Nomad v1.8.1
BuildDate 2024-06-19T06:43:57Z
Revision 5022543e4b7b8dcec9df123f86630ae3fdcffbe6
```
### Operating system and Environment details
```
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
```
### Issue
I runs hundreds of jobs a day on a machine and always set `resources.cores` for my jobs. Occasionally I get the below issue
```
2024-08-14 08:59:15.919991105 +0800 +08: Received - Task received by client
2024-08-14 08:59:15.923095086 +0800 +08: Setup Failure - failed to setup alloc: pre-run hook "cpuparts_hook" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: device or resource busy
```
Once this happens, all jobs on the machine fail to run including Docker and raw_exec jobs. This issue persists until I manually remove `/sys/fs/cgroup/nomad.slice/reserve.slice/`. I suspect that Nomad fails to clean up reserved cores under some unexpected failure circumstances. I tried removing `resources.cores` config and it's been working just fine.
### Reproduction steps
Not sure how to reproduce this.
#### Expected Result
No error when scheduling jobs.
#### Actual Result
```
2024-08-14 08:59:15.919991105 +0800 +08: Received - Task received by client
2024-08-14 08:59:15.923095086 +0800 +08: Setup Failure - failed to setup alloc: pre-run hook "cpuparts_hook" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: device or resource busy
```
### Job file (if appropriate)
### Nomad Server logs (if appropriate)
### Nomad Client logs (if appropriate)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.