Allocation GC fails silently
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
```
Nomad v2.0.0
BuildDate 2026-04-21T15:42:13Z
Revision 7a90601f4d9bc473718311fe5167c0c1f66500ba
```
### Operating system and Environment details
Debian
### Issue
In some occasions allocations are marked for GC and collected according to the log, but old data remains on disk:
`journalctl -u nomad | grep ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
2026-07-29T10:01:16.831+0200 [INFO] client.gc: marking allocation for GC: alloc_id=ea308738-40b6-f5c4-9f12-e89d46a9aa1c
2026-07-30T10:02:45.415+0200 [INFO] client.gc: garbage collecting allocation: alloc_id=ea308738-40b6-f5c4-9f12-e89d46a9aa1c reason="forced collection"
```
`tree /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
/opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c
├── alloc
│ ├── data
│ ├── logs
│ │ ├── foobar.stderr.0
│ │ └── foobar.stdout.0
│ └── tmp
└── foobar
├── local
├── private
│ └── vault_token
├── secrets
│ ├── foo_access_token
│ ├── bar.jwt
│ └── vault_token
└── tmp
```
`mount | grep ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
tmpfs on /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c/foobar/secrets type tmpfs (rw,noexec,relatime,size=1024k,inode64)
tmpfs on /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c/foobar/private type tmpfs (rw,noexec,relatime,size=1024k,inode64)
```
`du -hs /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
56K /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c
```
Those allocs result in a 404 if requested by API, but remain in the client state database:
`bbolt keys /opt/nomad/data/client/state.db allocations ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
acknowledged_state
alloc
alloc_consul_acl_token_identities
alloc_identities
task-foobar
```
This isn't limited to docker allocs but happens with the exec driver as well.
Might be similar to #20544.
### Reproduction steps
I'm not sure how to reproduce this yet. In other occasions the GC works just fine.
### Temporary mitigation
For now, we're periodically running a script that forcibly removes stale allocs if the disk or inode usage is over 80 percent. Requires a `NOMAD_TOKEN` with `read-job` capability on all namespaces.
```bash
#!/usr/bin/env bash
NOMAD_ADDR="http://127.0.0.1:4646"
ALLOC_DIR="/opt/nomad/data/alloc"
: "${NOMAD_TOKEN:?NOMAD_TOKEN env var required}"
requires_forced_gc=$(
df -H / | awk 'NR>1{gsub(/%/,"",$5); if($5+0>80) print "DISK "$5"% "$6}'
df -i / | awk 'NR>1{gsub(/%/,"",$5); if($5+0>80) print "INODE "$5"% "$6}'
)
[ -z "$requires_forced_gc" ] && exit 0
echo "Disk or inode usage >80%. Forcibly removing stale allocs."
for path in "$ALLOC_DIR"/*/; do
id=$(basename "$path")
[ "$id" = "*" ] && continue
code=$(curl -s -o /dev/null -w '%{http_code}' \
-H "X-Nomad-Token: $NOMAD_TOKEN" \
"$NOMAD_ADDR/v1/allocation/$id")
if [ "$code" = "404" ]; then
echo "Removing stale alloc $id"
grep "/opt/nomad/data/alloc/${id:?}" /proc/mounts | awk '{print $2}' \
| sort -r | xargs -r -n1 umount -l
if [ "$?" == 1 ]; then
echo "Failed to lazily detach the remaining alloc filesystems";
else
rm -Rf "${ALLOC_DIR:?}/${id:?}"
if [ "$?" == 1 ]; then
echo "Failed to remove the alloc data dir";
fi
fi
else
echo "Keeping $id (HTTP status code $code)"
fi
done
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.