Intermittent vault 403 errors when using workload identity
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
My client version:
```
Nomad v1.7.3
BuildDate 2024-01-15T16:55:40Z
Revision 60ee328f97d19d2d2d9761251b895b06d82eb1a1
```
Linux and Mac are using nomad v1.7.5
Windows are using nomad v1.7.2 due to #20034
### Operating system and Environment details
Ubuntu 22.04 amd64
Windows Server 2019 Datacenter amd64
Mac amd64 & arm64
### Issue
We are using nomad to run thousands of batch jobs per day. After converting to use workload identity with vault we are seeing intermittent 403 errors when a job tries to access vault, maybe 4-5 times per 1000 jobs. Retrying the job usually fixes the issue but is not a good solution for us due to increasing scale and how downstream systems work. This happens on all platforms.
```
Template failed: vault.read(kv/data//): vault.read(kv/data//): Error making API request. URL: GET https://active.vault.service.consul:8200/v1/kv/data// Code: 403. Errors: * 1 error occurred: * permission denied
```
### Reproduction steps
nomad server vault stanza:
```terraform
vault {
enabled = true
address = "https://active.vault.service.consul:8200"
ca_file = "/etc/vault.d/vault-agent-ca.pem"
cert_file = "/etc/vault.d/client-vault-cert.pem"
key_file = "/etc/vault.d/client-vault-key.pem"
create_from_role = "nomad-workload-id"
}
```
nomad client vault stanza:
```terraform
vault {
enabled = true
address = "https://active.vault.service.consul:8200"
ca_file = "/etc/vault.d/vault-agent-ca.pem"
cert_file = "/etc/vault.d/client-vault-cert.pem"
key_file = "/etc/vault.d/client-vault-key.pem"
create_from_role = "nomad-workload-id"
}
```
nomad-workload-id vault policy:
```terraform
path "kv/data/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_task}}/*" {
capabilities = ["read"]
}
path "kv/data/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_task}}" {
capabilities = ["read"]
}
path "kv/metadata/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_task}}/*" {
capabilities = ["list"]
}
path "kv/metadata/*" {
capabilities = ["list"]
}
```
#### Expected Result
Consistently able to access vault
#### Actual Result
Intermittently fails
### Job file (if appropriate)
Every job task has the same vault stanza:
```json
"Vault": {
"Role": "nomad-workload-id",
"Policies": null,
"Namespace": "",
"Cluster": "default",
"Env": true,
"DisableFile": false,
"ChangeMode": "restart",
"ChangeSignal": "SIGHUP",
"AllowTokenExpiration": false
}
```
and an identity like this:
```json
"Identities": [
{
"Name": "redacted",
"Audience": [
"redacted"
],
"ChangeMode": "",
"ChangeSignal": "",
"Env": false,
"File": false,
"ServiceName": "",
"TTL": 3600000000000
}
]
```
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is identified. Start by reproducing the intermittent failure with the Nomad server and client Vault stanzas, workload identity, and job configuration shown here; compare successful and failed Vault access attempts, with the goal of making access consistent without retrying jobs.
Written by the indexing model from the issue text.
Assessment
- Domain
- authentication
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100