hashicorp / hashicorp/nomad

Workload identity failures with vault secrets

Open
#20,104 1 comment 4 reactions 0 assignees View on GitHub
stage/needs-investigation theme/workload-identity type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Hello guys,

There is really something wrong when workload identity is enabled.
4 tries and 4 rollbacks:
- 2 because there was some bugs and they have been solved
- 1 my bad because I put the jwt path
- right jwt path because there is a remaining bug

We are successfully running 2 nomad clusters in 2 regions more than 2 years with vault legacy mode.

So after the 2nd rollback, I decided to spawn a new cluster to better debug what's going on and also not to impact our developers on our non production cluster.

I deployed 2 workloads with one instance for each of them and everything has been working for like 2 weeks without any reboot.

Then yesterday I decided to enable workload identity on our npd cluster and redeploy all applications. We have like 9 applications and most of them with multiple instances and multiple namespaces.

Everthing were up and running but in less than 30min things started to fail.
Reasons of the failure: Thousands of Vault 403 errors in nomad logs. I could obviously see errors on Vault side.

As the time was ticking I dediced to rollback as there was no obvious reason. nomad monitor in debug/trace mode wasn't helpful.

So today, I decided to add more workloads on our testing cluster with a dummy application with vault secrets.
4 namespaces with 1 deloyment in each and 10 instances. So in total 40 nomad jobs.
Everything were fine but when 30min started approched, things went wrong again just like our npd cluster.

nomad servers config:
```
vault {
enabled = true
ca_path = "/etc/nomad.d/vault-ca.crt"
address = "https://REDACTED:8200"
default_identity {
aud = ["vault.io"]
env = false
file = true
ttl = "1h"
}
}
```

nomad clients config:
```
vault {
enabled = true
ca_path = "/etc/nomad.d/vault-ca.crt"
address = "https://REDACTED:8200"
jwt_auth_backend_path = "xxxx/jwt/npd/nomad/euw1"
}
```

Normally the jwt token renewal is happenning around 30min.

On our oss Vault cluster, I could see thousands of lines like:

```
Mar 08 11:16:46 xxxx vault[1538080]: 2024-03-07T22:16:46.221Z [INFO] expiration: revoked lease: lease_id=auth/xxxx/jwt/npd/nomad/euw1/login/hc9bb76e7c451850901bae1291e6997e2555e1560f71cd3639198fb13fc05fb7c

Mar 08 11:19:47 xxxx vault[1538080]: 2024-03-07T22:19:47.343Z [INFO] http: TLS handshake error from 10.200.0.105:52580: EOF
```

`10.200.0.105` is a nomad client.

nomad logs:
```
2024-03-06T11:20:56.442Z [WARN] agent: (view) vault.read(secret/data/REDACTED): vault.read(secret/data/REDACTED): Error making API request.

URL: GET https://REDACTED.fr:8200/v1/secret/data/REDACTED
Code: 403. Errors:

* 1 error occurred:
* permission denied

(retry attempt 7 after "16s")
```

Even if our legacy mode was perfectly working, I try to put `tls_skip_verify` to true but still no luck.

Somehow, when you use many namespaces things get really wrong.
I don't know if/how it's related or the number of deployments, but I purged all deployments on my testing cluster and reinstall the same dummy application in only one namespace. So 1 namespace with 1 deloyment and 10 instances.

Guess what, everything is fine with no reboot at all. As the time of writing this issue, the dummy app has been up for almost 1 hour.

Nomad version:
```
Nomad v1.7.5+ent
BuildDate 2024-02-13T15:44:21Z
Revision a6a9c85c31897ab7cb1159d9b1cae0648b8620ca
```

Vault version:
```
Vault v1.13.13 (12c68983303db23056dfd4295b06724310de3fec), built 2024-01-29T17:48:08Z
```

Consul version:
```
Consul v1.17.3+ent
Revision 7dcfe6a3
Build Date 2024-02-13T18:58:23Z
```

### Reproduction steps

dummy app:
```hcl
job "fake" {
region = "europe-west1"
datacenters = ["euw1"]
type = "service"
namespace = "[[.namespace]]"

update {
max_parallel = 1
health_check = "checks"
min_healthy_time = "20s"
healthy_deadline = "2m"
canary = 1
auto_revert = true
auto_promote = true
}

group "application" {
count = 10

network {
mode = "bridge"
port "http" {
to = 80
}
port "envoy_metrics" {
to = 9102
}
}

consul {
namespace = "[[.namespace]]"
}

service {
name = "fake"
port = "80"
tags = ["fake"]
check {
expose = true
type = "http"
path = "/"
interval = "10s"
timeout = "2s"
}

meta {
envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}"
region = "${meta.region}"
zone = "${meta.zone}"
}

connect {
sidecar_service {
proxy {
expose {
path {
path = "/metrics"
protocol = "http"
local_path_port = 9101
listener_port = "http"
}
}
}
}
}
}

task "fake" {
vault {}

driver = "docker"

config {
ports = ["http"]
image = "nginx:1.25.4"

}

resources {
cpu = 64
memory = 64
}

template {
data = <

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.