hashicorp / hashicorp/vault-secrets-operator
Kubernetes auth method does not work with "Use client JWT as reviewer JWT" option - with workaround
- Dominant language
- Go
- Stars
- 600
- Forks
- 146
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 6
Description
**Describe the bug**
One of the options for configuring the Kubernetes auth method described in [Kubernetes auth method](https://developer.hashicorp.com/vault/docs/auth/kubernetes) is "Use client JWT as reviewer JWT". This does not work out-of-the-box with VSO and will result in a HTTP 403 error for `PUT https://{VAULT_ADDR}/v1/auth/{MOUNT_PATH}` . This is because Vault attempts to authenticate to the Kubernetes TokenReview API with the client JWT, but this JWT is missing the appropriate audience claim for the Kubernetes APIServer.
**To Reproduce**
Steps to reproduce the behavior:
1. Configure the Kubernetes Auth backend according to the supplied Terraform example
2. Deploy VSO using the supplied values.yaml
3. Apply the supplied CustomResources
4. Run `kubectl describe VaultDynamicSecret app-db-creds -n demo-ns` to see the following error:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning VaultClientConfigError 1s (x7 over 1s) VaultDynamicSecret Failed to get Vault client: Error making API request.
URL: PUT https://{VAULT_ADDR}/v1/auth/kubernetes/login
Code: 403. Errors:
* permission denied, lease_id=
```
**Application deployment:**
Auth config in Terraform:
```
resource "vault_auth_backend" "kubernetes" {
type = "kubernetes"
path = "kubernetes"
}
resource "vault_kubernetes_auth_backend_config" "kubernetes_auth" {
backend = vault_auth_backend.kubernetes.path
kubernetes_host = one(data.azurerm_kubernetes_cluster.aks.kube_config).host
kubernetes_ca_cert = base64decode(
one(data.azurerm_kubernetes_cluster.aks.kube_config).cluster_ca_certificate
)
disable_local_ca_jwt = true
}
```
`values.yaml`
```yaml
defaultVaultConnection:
enabled: true
address: https://{VAULT_ADDR}
```
CustomResources:
```
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: app
namespace: demo-ns
spec:
kubernetes:
role: demo-ns-app
serviceAccount: default
tokenExpirationSeconds: 600
audiences:
- vault
method: kubernetes
mount: kubernetes
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
name: app-db-creds
namespace: demo-ns
spec:
destination:
name: app-db-creds
create: true
mount: database
path: creds/app-dbuser
rolloutRestartTargets:
- kind: Deployment
name: app
vaultAuthRef: app
```
Other useful info to include: `kubectl describe deployment ` and `kubectl describe ` output.
**Expected behavior**
When VSO generates the JWT for the `ServiceAccount` it should (by default) include the Kubernetes APIServer audience for the cluster in the `aud` claim. The VaultDynamicSecret will then be created as expected.
Alternatively, this behavior could be configured by an additional parameter in the `VaultAuth` CRD to tell VSO that the client JWT is being used as the token reviewer JWT.
**Environment**
* Kubernetes version: 1.26.3
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): AKS
* Other configuration options or runtime services (istio, etc.): istio 1.18.2
* vault-secrets-operator version: 0.2.0
**Additional context**
**Workaround**
For now, you can resolve this by manually adding the APIServer audience to the `VaultAuth` CustomResource. For example, for Azure Kubernetes Service this looks like:
```
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: app
namespace: demo-ns
spec:
kubernetes:
role: demo-ns-app
serviceAccount: default
tokenExpirationSeconds: 600
audiences:
- vault
- "https://{REDACTED}.hcp.westeurope.azmk8s.io"
method: kubernetes
mount: kubernetes
```
If you are unsure of the APIServer audience for your cluster, use one of [these examples](https://gist.github.com/angelo-v/e0208a18d455e2e6ea3c40ad637aac53) to decode the JWT mounted in `/run/secrets/kubernetes.io/serviceaccount/token` for an existing pod and inspect the contents of the `aud` claim.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the VaultAuth CRD's Kubernetes authentication settings and reproduce the failure using the Terraform backend configuration, supplied values.yaml, and CustomResources. Inspect the JWT audience used for the Kubernetes TokenReview API and the kubectl describe VaultDynamicSecret output. Done means the documented client-JWT reviewer configuration works without a manual audience workaround, or the required VaultAuth option is clearly supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- authentication, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100