hashicorp / hashicorp/nomad

Allocation GC fails silently

Open
#28,467 3 comments 0 reactions 1 assignee Claimed by @tgross View on GitHub
stage/waiting-reply theme/client type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
```
Nomad v2.0.0
BuildDate 2026-04-21T15:42:13Z
Revision 7a90601f4d9bc473718311fe5167c0c1f66500ba
```

### Operating system and Environment details

Debian

### Issue

In some occasions allocations are marked for GC and collected according to the log, but old data remains on disk:

`journalctl -u nomad | grep ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
2026-07-29T10:01:16.831+0200 [INFO] client.gc: marking allocation for GC: alloc_id=ea308738-40b6-f5c4-9f12-e89d46a9aa1c
2026-07-30T10:02:45.415+0200 [INFO] client.gc: garbage collecting allocation: alloc_id=ea308738-40b6-f5c4-9f12-e89d46a9aa1c reason="forced collection"
```

`tree /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
/opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c
├── alloc
│   ├── data
│   ├── logs
│   │   ├── foobar.stderr.0
│   │   └── foobar.stdout.0
│   └── tmp
└── foobar
├── local
├── private
│   └── vault_token
├── secrets
│   ├── foo_access_token
│   ├── bar.jwt
│   └── vault_token
└── tmp
```

`mount | grep ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
tmpfs on /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c/foobar/secrets type tmpfs (rw,noexec,relatime,size=1024k,inode64)
tmpfs on /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c/foobar/private type tmpfs (rw,noexec,relatime,size=1024k,inode64)
```

`du -hs /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
56K /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c
```

Those allocs result in a 404 if requested by API, but remain in the client state database:
`bbolt keys /opt/nomad/data/client/state.db allocations ea308738-40b6-f5c4-9f12-e89d46a9aa1c`
```
acknowledged_state
alloc
alloc_consul_acl_token_identities
alloc_identities
task-foobar
```

This isn't limited to docker allocs but happens with the exec driver as well.

Might be similar to #20544.

### Reproduction steps

I'm not sure how to reproduce this yet. In other occasions the GC works just fine.

### Temporary mitigation

For now, we're periodically running a script that forcibly removes stale allocs if the disk or inode usage is over 80 percent. Requires a `NOMAD_TOKEN` with `read-job` capability on all namespaces.

```bash
#!/usr/bin/env bash

NOMAD_ADDR="http://127.0.0.1:4646"
ALLOC_DIR="/opt/nomad/data/alloc"

: "${NOMAD_TOKEN:?NOMAD_TOKEN env var required}"

requires_forced_gc=$(
df -H / | awk 'NR>1{gsub(/%/,"",$5); if($5+0>80) print "DISK "$5"% "$6}'
df -i / | awk 'NR>1{gsub(/%/,"",$5); if($5+0>80) print "INODE "$5"% "$6}'
)

[ -z "$requires_forced_gc" ] && exit 0

echo "Disk or inode usage >80%. Forcibly removing stale allocs."

for path in "$ALLOC_DIR"/*/; do
id=$(basename "$path")
[ "$id" = "*" ] && continue

code=$(curl -s -o /dev/null -w '%{http_code}' \
-H "X-Nomad-Token: $NOMAD_TOKEN" \
"$NOMAD_ADDR/v1/allocation/$id")

if [ "$code" = "404" ]; then
echo "Removing stale alloc $id"
grep "/opt/nomad/data/alloc/${id:?}" /proc/mounts | awk '{print $2}' \
| sort -r | xargs -r -n1 umount -l
if [ "$?" == 1 ]; then
echo "Failed to lazily detach the remaining alloc filesystems";
else
rm -Rf "${ALLOC_DIR:?}/${id:?}"
if [ "$?" == 1 ]; then
echo "Failed to remove the alloc data dir";
fi
fi
else
echo "Keeping $id (HTTP status code $code)"
fi
done
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.