hashicorp / hashicorp/nomad

Nomad fails to clean up reserved cores

Open
#23,965 10 comments 1 reaction 0 assignees View on GitHub
stage/needs-investigation theme/client theme/platform-windows type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Output from `nomad version`

```
Nomad v1.8.1
BuildDate 2024-06-19T06:43:57Z
Revision 5022543e4b7b8dcec9df123f86630ae3fdcffbe6
```

### Operating system and Environment details

```
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
```

### Issue

I runs hundreds of jobs a day on a machine and always set `resources.cores` for my jobs. Occasionally I get the below issue

```
2024-08-14 08:59:15.919991105 +0800 +08: Received - Task received by client
2024-08-14 08:59:15.923095086 +0800 +08: Setup Failure - failed to setup alloc: pre-run hook "cpuparts_hook" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: device or resource busy
```

Once this happens, all jobs on the machine fail to run including Docker and raw_exec jobs. This issue persists until I manually remove `/sys/fs/cgroup/nomad.slice/reserve.slice/`. I suspect that Nomad fails to clean up reserved cores under some unexpected failure circumstances. I tried removing `resources.cores` config and it's been working just fine.

### Reproduction steps

Not sure how to reproduce this.

#### Expected Result

No error when scheduling jobs.

#### Actual Result
```
2024-08-14 08:59:15.919991105 +0800 +08: Received - Task received by client
2024-08-14 08:59:15.923095086 +0800 +08: Setup Failure - failed to setup alloc: pre-run hook "cpuparts_hook" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: device or resource busy
```

### Job file (if appropriate)

### Nomad Server logs (if appropriate)

### Nomad Client logs (if appropriate)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.