hashicorp / hashicorp/nomad

WI: Intermittent expired consul tokens after upgrade

Open
#25,898 3 comments 1 reaction 0 assignees View on GitHub
hcc/jira stage/needs-investigation theme/consul theme/workload-identity type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Output from `nomad version`

v1.9.7

### Operating system and Environment details

Debian 12 Linux

### Issue

After enabling Consul Workload Identities, allocations apparently randomly fail with error:

```
failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for service sample-http: Unexpected response code: 500 (rpc error making call: error validating claims: go-jose/go-jose/jwt: validation failed, token is expired (exp))
```

After failing, scheduling will usually retry and succeed.

For consul connect tasks, this can also affect sidecars.

### Reproduction steps

- Set up Consul ACL for WI based on https://github.com/hashicorp-modules/terraform-consul-nomad-setup.
- In particular, there is no `ttl` set for the auth method

#### Expected Result

Jobs schedule successfully consistently

#### Actual Result

Allocations randomly fail. Can happen at initial scheduling attempt and is also triggered by restarting the nomad process (meaning restarting nomad process results in loss of availability of affected running workloads).

### Job file (if appropriate)

```
job "sample" {
group "sample" {
network {
port "http" {
static = 8080
}
}
consul {}
service {
name = "sample-http"
port = "http"
identity {
aud = ["consul.io"]
}
}
task "sample" {
identity {
aud = ["consul.io", "vault.io"]
}
driver = "docker"
config {
image = "localhost/sample"
}
template {
data = "{{ with secret "consul/creds/sample" }}{{.Data.token}}{{end}}"
destination = "secrets/ctoken"
}

}
}
}
```

### Nomad Server logs (if appropriate)

### Nomad Client logs (if appropriate)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with Nomad v1.9.7, the provided Consul ACL setup, and the sample job. Trace the Consul workload-identity token derivation and expiration validation described in the error. Done means allocations schedule consistently and restarting Nomad does not cause affected workloads to fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
authentication, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.