hashicorp / hashicorp/vault-helm
The Vault chart should expose a service excluding non-ready pods for client applications
- Dominant language
- Shell
- Stars
- 1.3k
- Forks
- 898
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe.**
In one of our deployments, we have a HA cluster with 3 Vault instances. Those instances are using a transit seal.
We recently had an incident; when the vault token used by the seal expired (our bad), one of the vault instance restarted and it remained unsealed due to the expired token. Due to the inclusion of non-ready pods, it immediately resulted in a 30-40% error rate.
**Describe the solution you'd like**
A service should be created by the chart for the use of client applications, so that only "ready" Vault instances would be targeted.
I understand non-ready pods are necessary for cluster join operations, but ideally, a separate service should be used for this purpose and for vault clients.
**Describe alternatives you've considered**
Implementing retries only slightly mitigates this issue due to the high error rate, and the lack of control over the round-robin client load balancing by default.
We ended up defining an additional "service" to exclude non-ready pods.
**Additional context**
While we encountered this issue in the case of an expired token, this could affected any failing restart of a vault instance or even while vault is starting.
We could have lived happily until the working day with 2/3 instances.
Contributor guide
Research direction
Start by reading the Vault Helm chart's service-related templates and values, then trace how readiness is currently represented for Vault pods. Compare the existing service behavior with the requested client-facing service. Done means client applications target only ready Vault instances while cluster join operations retain their required behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100