hashicorp / hashicorp/vault-helm

The Vault chart should expose a service excluding non-ready pods for client applications

Open
#430 1 comment 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe.**
In one of our deployments, we have a HA cluster with 3 Vault instances. Those instances are using a transit seal.

We recently had an incident; when the vault token used by the seal expired (our bad), one of the vault instance restarted and it remained unsealed due to the expired token. Due to the inclusion of non-ready pods, it immediately resulted in a 30-40% error rate.

**Describe the solution you'd like**
A service should be created by the chart for the use of client applications, so that only "ready" Vault instances would be targeted.

I understand non-ready pods are necessary for cluster join operations, but ideally, a separate service should be used for this purpose and for vault clients.

**Describe alternatives you've considered**
Implementing retries only slightly mitigates this issue due to the high error rate, and the lack of control over the round-robin client load balancing by default.

We ended up defining an additional "service" to exclude non-ready pods.

**Additional context**
While we encountered this issue in the case of an expired token, this could affected any failing restart of a vault instance or even while vault is starting.

We could have lived happily until the working day with 2/3 instances.

Contributor guide

Open the contributing guide

Research direction

Start by reading the Vault Helm chart's service-related templates and values, then trace how readiness is currently represented for Vault pods. Compare the existing service behavior with the requested client-facing service. Done means client applications target only ready Vault instances while cluster join operations retain their required behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm
Domain
devops, infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.