hashicorp / hashicorp/nomad

Intermittent vault 403 errors when using workload identity

Open
#20,099 0 comments 1 reaction 0 assignees View on GitHub
stage/needs-investigation theme/workload-identity type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

My client version:

```
Nomad v1.7.3
BuildDate 2024-01-15T16:55:40Z
Revision 60ee328f97d19d2d2d9761251b895b06d82eb1a1
```

Linux and Mac are using nomad v1.7.5

Windows are using nomad v1.7.2 due to #20034

### Operating system and Environment details

Ubuntu 22.04 amd64
Windows Server 2019 Datacenter amd64
Mac amd64 & arm64

### Issue

We are using nomad to run thousands of batch jobs per day. After converting to use workload identity with vault we are seeing intermittent 403 errors when a job tries to access vault, maybe 4-5 times per 1000 jobs. Retrying the job usually fixes the issue but is not a good solution for us due to increasing scale and how downstream systems work. This happens on all platforms.

```
Template failed: vault.read(kv/data//): vault.read(kv/data//): Error making API request. URL: GET https://active.vault.service.consul:8200/v1/kv/data// Code: 403. Errors: * 1 error occurred: * permission denied
```

### Reproduction steps

nomad server vault stanza:

```terraform
vault {
enabled = true
address = "https://active.vault.service.consul:8200"
ca_file = "/etc/vault.d/vault-agent-ca.pem"
cert_file = "/etc/vault.d/client-vault-cert.pem"
key_file = "/etc/vault.d/client-vault-key.pem"
create_from_role = "nomad-workload-id"
}
```

nomad client vault stanza:

```terraform
vault {
enabled = true
address = "https://active.vault.service.consul:8200"
ca_file = "/etc/vault.d/vault-agent-ca.pem"
cert_file = "/etc/vault.d/client-vault-cert.pem"
key_file = "/etc/vault.d/client-vault-key.pem"
create_from_role = "nomad-workload-id"
}
```

nomad-workload-id vault policy:

```terraform
path "kv/data/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_task}}/*" {
capabilities = ["read"]
}

path "kv/data/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_task}}" {
capabilities = ["read"]
}

path "kv/metadata/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_ebe43278.metadata.nomad_task}}/*" {
capabilities = ["list"]
}

path "kv/metadata/*" {
capabilities = ["list"]
}
```

#### Expected Result

Consistently able to access vault

#### Actual Result

Intermittently fails

### Job file (if appropriate)

Every job task has the same vault stanza:

```json
"Vault": {
"Role": "nomad-workload-id",
"Policies": null,
"Namespace": "",
"Cluster": "default",
"Env": true,
"DisableFile": false,
"ChangeMode": "restart",
"ChangeSignal": "SIGHUP",
"AllowTokenExpiration": false
}
```

and an identity like this:

```json
"Identities": [
{
"Name": "redacted",
"Audience": [
"redacted"
],
"ChangeMode": "",
"ChangeSignal": "",
"Env": false,
"File": false,
"ServiceName": "",
"TTL": 3600000000000
}
]
```

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is identified. Start by reproducing the intermittent failure with the Nomad server and client Vault stanzas, workload identity, and job configuration shown here; compare successful and failed Vault access attempts, with the goal of making access consistent without retrying jobs.

Written by the indexing model from the issue text.

Assessment

Domain
authentication
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.