hashicorp / hashicorp/nomad

Batch jobs using vault fail with error "unable to find token for workload"

Open
#25,955 11 comments 4 reactions 0 assignees View on GitHub
hcc/jira theme/vault theme/workload-identity type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
```
Nomad v1.10.1
BuildDate 2025-05-13T07:40:43Z
Revision 3431f13e8036b4716aac0e3b8c5854ddca212e5c
```

### Operating system and Environment details
Debian GNU/Linux 13 (trixie) x86_64, 6.9.9-amd64
Nomad installed through apt: nomad/bookworm,now 1.10.1-1 amd64

### Issue
Unable to run any periodic/batch jobs that use Vault due to weird error:
```
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro type=Received msg="Task received by client" failed=false
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro type="Task Setup" msg="Building Task Directory" failed=false
[ERROR] client.alloc_runner.task_runner.task_hook.vault: failed to derive Vault token: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro error="failed to retrieve signed workload identity: unable to find token for workload \"periodic-repro\" and identity \"vault_default\"" recoverable=true backoff=5s
[ERROR] client.alloc_runner.task_runner.task_hook.vault: failed to derive Vault token: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro error="failed to retrieve signed workload identity: unable to find token for workload \"periodic-repro\" and identity \"vault_default\"" recoverable=true backoff=10s
[ERROR] client.alloc_runner.task_runner.task_hook.vault: failed to derive Vault token: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro error="failed to retrieve signed workload identity: unable to find token for workload \"periodic-repro\" and identity \"vault_default\"" recoverable=true backoff=20s
```
From what I can tell the error comes from here, perhaps it is related to the mentioned race condition? https://github.com/hashicorp/nomad/blob/348177d118a129cdc9196b6f8cb0181caac8b41b/client/widmgr/widmgr.go#L147-L149

This is happening with all the periodic jobs I have and seem to happen regardless of what the job spedcification looks like, as long as it contains a `vault` section. The issue appeared without me consiously changing anything in Nomad, Vault, or the job specification, as the job just started failing on their own.

### Reproduction steps
1. Submit the job to nomad
2. Force launch a periodic job
3. Job should be stuck in either pending or recovering (depending on how long it is left alone) state with the error above in the nomad logs

#### Expected Result
Job starts and runs without issue.

#### Actual Result
Started job (and any future instances of the periodic job) are stuck in pending state forever with the mentioned error in the nomad logs.

### Job file (if appropriate)
Reproducible with this minimal job:
```hcl
job "periodic-repro" {
region = "dk"
datacenters = ["dk1"]
type = "batch"
namespace = "monitor"

periodic {
cron = "0 0 * * *"
}

group "periodic-repro" {
count = 1

task "periodic-repro" {
driver = "docker"

config {
image = "busybox:latest"
network_mode = "host"
command = "sh"
args = ["-c", "echo 'Hello World' && sleep 120"]
}

resources {
cpu = 100
memory = 256
}

vault {
policies = ["monitor-nrgi"]
change_mode = "restart"
}
}
}
}
```

### Nomad Server logs (if appropriate)
These are the logs from my nomad server which is also the client the job runs on:
```
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro type=Received msg="Task received by client" failed=false
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro type="Task Setup" msg="Building Task Directory" failed=false
[ERROR] client.alloc_runner.task_runner.task_hook.vault: failed to derive Vault token: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro error="failed to retrieve signed workload identity: unable to find token for workload \"periodic-repro\" and identity \"vault_default\"" recoverable=true backoff=5s
[ERROR] client.alloc_runner.task_runner.task_hook.vault: failed to derive Vault token: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro error="failed to retrieve signed workload identity: unable to find token for workload \"periodic-repro\" and identity \"vault_default\"" recoverable=true backoff=10s
[ERROR] client.alloc_runner.task_runner.task_hook.vault: failed to derive Vault token: alloc_id=91a315b9-dfc6-0353-fe7f-d0e2e540a323 task=periodic-repro error="failed to retrieve signed workload identity: unable to find token for workload \"periodic-repro\" and identity \"vault_default\"" recoverable=true backoff=20s
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at client/widmgr/widmgr.go around lines 147-149 and trace how periodic batch allocations retrieve signed workload identities for Vault. Reproduce the failure with the minimal periodic job, then verify that it starts and runs without the missing-workload-token error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.