hashicorp / hashicorp/vault-helm

The vault service in HA mode with raft storage does not support loadbalancing.

Open
#457 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Describe the bug**
The Vault service in HA mode does not support loadbalancing.

**To Reproduce**
Steps to reproduce the behavior:
1. Install chart in HA mode on an HA Kubernetes Cluster
2. Configure Istio properly (an entirely separate set of issues not related to this issue...see Istio sidecar annotations in the values file), and create a VirtualService for Vault.
3. `curl` or use the vault cli to enable raft snapshots

**Expected behavior**
HA capability to interact with raft storage.

**Environment**
* Kubernetes version: RKE2
* chart version 0.8.0

Chart values:
(vault config)
```yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: vault
namespace: vault
spec:
targetNamespace: vault
releaseName: vault
interval: 10m
chart:
spec:
chart: chart
sourceRef:
kind: GitRepository
name: vault
install:
remediation:
retries: 5
upgrade:
remediation:
retries: 5
remediateLastFailure: true
cleanupOnFail: true
rollback:
timeout: 10m
cleanupOnFail: false
values:

# Vault Helm Chart Value Overrides
global:
enabled: true
tlsDisable: true
imagePullSecrets:
- name: private-registry

injector:
enabled: false
# Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
image:
repository: "registry1.dso.mil/ironbank/hashicorp/vault/vault-k8s"
tag: v0.6.0

resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m

server:

dataStorage:
enabled: true
size: 50Gi
mountPath: "/vault/data"
accessMode: ReadWriteOnce

postStart:
- /bin/sh
- -c
- 'echo "libevmulti_init: Ready " > /tmp/cloudhsm_client_start.log'

# the following annotations are requied to exempt traffic from the raft protocol
# otherwise, envoy side-car will interrupt raft-member to raft-member tls communications
annotations:
traffic.sidecar.istio.io/excludeInboundPorts: "8201"
traffic.sidecar.istio.io/excludeOutboundPorts: "8201"
image:
# Enterprise Image - license required
repository: "registry1.dsop.io/ironbank/hashicorp/turbog"
tag: "1.6.1-hsm-cloudhsm"
# tag: "1.5.0_ent"

#IB-Enterprise
# repository: "registry1.dsop.io/ironbank/hashicorp/secure-secrets-management/vault-enterprise"
# tag: "1.5.3"


# The Following Resource Limits are in line with node requirements in the
# Vault Reference Architecture for a Small Cluster

# PROD
resources:
requests:
memory: 8Gi
cpu: 2000m
limits:
memory: 16Gi
cpu: 2000m

ingress:
enabled: false

# For HA configuration and because we need to manually init the vault,
# we need to define custom readiness/liveness Probe settings
readinessProbe:
enabled: false
path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204" #orinal setting
livenessProbe:
enabled: false
path: "/v1/sys/health?standbyok=true"
initialDelaySeconds: 60

# extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
# used to include variables required for auto-unseal.
extraEnvironmentVars:
VAULT_API_ADDR: http://vault-internal:8200
VAULT_ADDR: http://127.0.0.1:8200

# This configures the Vault Statefulset to create a PVC for audit logs.
# See https://www.vaultproject.io/docs/audit/index.html to know more
auditStorage:
enabled: true

# Run Vault in "HA" mode.
ha:
enabled: true
replicas: 5

raft:
enabled: true
setNodeId: true

# config file encrypted in vault-values.enc.yaml
config: |
ui = true
disable_mlock = true
log_level = "Trace"

listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_disable = 1
}

storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "http://vault-0.vault-internal:8200"
}
retry_join {
leader_api_addr = "http://vault-1.vault-internal:8200"
}
retry_join {
leader_api_addr = "http://vault-2.vault-internal:8200"
}
retry_join {
leader_api_addr = "http://vault-3.vault-internal:8200"
}
retry_join {
leader_api_addr = "http://vault-4.vault-internal:8200"
}
}

seal "awskms" {
region = "us-gov-west-1"
kms_key_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
endpoint = "https://kms.us-gov-west-1.amazonaws.com"
}

entropy "seal" {
mode = "augmentation"
}

service_registration "kubernetes" {}

# Vault UI
ui:
enabled: true
serviceType: "ClusterIP"
serviceNodePort: null
externalPort: 8200
```

**Additional context**

The Istio VirtualService

```yaml
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{},"labels":{"kustomize.toolkit.fluxcd.io/checksum":"10e40553b58e53b5e14ee98481ade71bbd2a77d6","kustomize.toolkit.fluxcd.io/name":"vault-deploy","kustomize.toolkit.fluxcd.io/namespace":"vault","owner":"vault"},"name":"vault","namespace":"vault"},"spec":{"gateways":["main.istio-system.svc.cluster.local"],"hosts":["our.vault.url"],"http":[{"route":[{"destination":{"host":"vault-active.vault.svc.cluster.local","port":{"number":8200}}}]}]}}
creationTimestamp: "2021-01-25T20:52:29Z"
generation: 3
labels:
kustomize.toolkit.fluxcd.io/checksum: 10e40553b58e53b5e14ee98481ade71bbd2a77d6
kustomize.toolkit.fluxcd.io/name: vault-deploy
kustomize.toolkit.fluxcd.io/namespace: vault
owner: vault
managedFields:
- apiVersion: networking.istio.io/v1beta1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:kustomize.toolkit.fluxcd.io/checksum: {}
f:kustomize.toolkit.fluxcd.io/name: {}
f:kustomize.toolkit.fluxcd.io/namespace: {}
f:owner: {}
f:spec:
.: {}
f:gateways: {}
f:hosts: {}
f:http: {}
manager: kubectl-client-side-apply
operation: Update
time: "2021-01-25T20:52:29Z"
name: vault
namespace: vault
resourceVersion: "8111567"
selfLink: /apis/networking.istio.io/v1beta1/namespaces/vault/virtualservices/vault
uid: 22c8d5af-ab80-470a-9c83-0034e65d41a2
spec:
gateways:
- main.istio-system.svc.cluster.local
hosts:
- our.sensitive.domain
http:
- route:
- destination:
host: vault-active.vault.svc.cluster.local
port:
number: 8200
kind: List
metadata:
resourceVersion: ""
selfLink: ""
```

Once the virtual service was changed to point to `vault-active` everything worked as expected, but we have 4 stale pods, which isn't really HA anymore.

Contributor guide

Open the contributing guide

Research direction

Start with the chart's HA and raft values and the Istio VirtualService routing to vault-active; compare the generated services and endpoints for the five replicas. Reproduce the raft snapshot interaction through the configured endpoint and verify that HA traffic reaches the intended Vault pods rather than leaving four pods unused.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.