WI: Intermittent expired consul tokens after upgrade
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
v1.9.7
### Operating system and Environment details
Debian 12 Linux
### Issue
After enabling Consul Workload Identities, allocations apparently randomly fail with error:
```
failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for service sample-http: Unexpected response code: 500 (rpc error making call: error validating claims: go-jose/go-jose/jwt: validation failed, token is expired (exp))
```
After failing, scheduling will usually retry and succeed.
For consul connect tasks, this can also affect sidecars.
### Reproduction steps
- Set up Consul ACL for WI based on https://github.com/hashicorp-modules/terraform-consul-nomad-setup.
- In particular, there is no `ttl` set for the auth method
#### Expected Result
Jobs schedule successfully consistently
#### Actual Result
Allocations randomly fail. Can happen at initial scheduling attempt and is also triggered by restarting the nomad process (meaning restarting nomad process results in loss of availability of affected running workloads).
### Job file (if appropriate)
```
job "sample" {
group "sample" {
network {
port "http" {
static = 8080
}
}
consul {}
service {
name = "sample-http"
port = "http"
identity {
aud = ["consul.io"]
}
}
task "sample" {
identity {
aud = ["consul.io", "vault.io"]
}
driver = "docker"
config {
image = "localhost/sample"
}
template {
data = "{{ with secret "consul/creds/sample" }}{{.Data.token}}{{end}}"
destination = "secrets/ctoken"
}
}
}
}
```
### Nomad Server logs (if appropriate)
### Nomad Client logs (if appropriate)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure with Nomad v1.9.7, the provided Consul ACL setup, and the sample job. Trace the Consul workload-identity token derivation and expiration validation described in the error. Done means allocations schedule consistently and restarting Nomad does not cause affected workloads to fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- authentication, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100