cpuset: `no space left on device`

### Nomad version

```
Nomad v1.7.7
BuildDate 2024-04-16T19:26:43Z
Revision 0f34c85ee63f6472bd2db1e2487611f4b176c70c
```

### Operating system and Environment details

Running Ubuntu 22.04 on Google Cloud in an n2d-standard-32 instance.

```
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```

### Issue

Alerts fired due to failed allocations. Upon investigation, I noticed the following log line:

```
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:58:45.224880+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
```

Also interesting to observe is that, unlike in our other 1.7.x clients, there's overlap between the CPUs for the reserve and share slices:

```
$ head /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus /sys/fs/cgroup/nomad.slice/share.slice/cpuset.cpus
==> /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus <==
0-3

==> /sys/fs/cgroup/nomad.slice/share.slice/cpuset.cpus <==
0-31
```

### Reproduction steps

Not clear how to reproduce. This happened on a single instance. All allocations that failed are from periodic jobs, running with on the `exec` driver with no `core` constraints.

#### Expected Result

Allocations spawn successfully.

#### Actual Result

Allocations failed to spawn.

### Nomad Client logs (if appropriate)

```json
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:56:52.170066+02:00","alloc_id":"7a39078a-0769-d3e9-38a0-c706ff516de8","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:58:45.224880+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:58:45.224934+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:58:45.226936+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:58:45.228745+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"processing","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:58:45.241815+02:00","alloc_id":"84407c34-35e1-b0cb-4a8f-f3b6c9a8cc81","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:59:45.699245+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:59:45.699331+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"processing","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:59:45.701453+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T09:59:45.703532+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T09:59:45.717471+02:00","alloc_id":"c506511a-eb17-73ee-7164-aaa9df390f47","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.588677+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.588716+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"maintenance-timer","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.594967+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.599704+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.617996+02:00","alloc_id":"69725229-e961-9496-917c-d81b21aac9d8","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.944803+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.944850+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.950686+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:00.954145+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:00.971800+02:00","alloc_id":"ac468e4e-a551-2991-a4fc-b8fa64599552","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.280014+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.280062+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"maintenance-timer","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.284932+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.290230+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.305828+02:00","alloc_id":"4b3df1d6-87a0-b0f6-38c6-b4e305d1f3da","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.604525+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.604609+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.609268+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:00:01.615208+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:00:01.625727+02:00","alloc_id":"7fa1997a-9aa0-50cf-bb01-85168b5e7ba0","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:01:00.615323+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:01:00.615380+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:01:00.617829+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:01:00.619727+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:01:00.633596+02:00","alloc_id":"727d4eeb-b135-3a31-1765-133146f2cd7f","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"error","@message":"prerun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:02:00.614056+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","error":"pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:02:00.614102+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"timer","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:02:00.615991+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"nix-setup-profiles","type":"Setup Failure"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-21T10:02:00.617758+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","failed":true,"msg":"failed to setup alloc: pre-run hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device","task":"promtail","type":"Setup Failure"}
{"@level":"error","@message":"postrun failed","@module":"client.alloc_runner","@timestamp":"2024-06-21T10:02:00.629413+02:00","alloc_id":"e1935c49-11a0-579c-5ab5-244bc931f0ad","error":"hook \"cpuparts_hook\" failed: write /sys/fs/cgroup/nomad.slice/reserve.slice/cpuset.cpus: no space left on device"}
```

### Nomad client configuration

```hcl
client {
gc_max_allocs = 300
gc_disk_usage_threshold = 80
}
```

Start by tracing the cpuparts_hook entry point and the client.alloc_runner prerun and postrun paths that write the cgroup cpuset.cpus files. Investigate the reserve.slice/share.slice overlap on Ubuntu 22.04 and determine why the write reports no space left on device. Done means allocations spawn successfully without the cpuset error or unexpected overlap.

Written by the indexing model from the issue text.