hashicorp / hashicorp/vault-helm

Support deploying central Vault Agent HTTP Caching Proxy

Open
#756 4 comments 9 reactions 0 assignees View on GitHub
enhancement
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe.**

Many Dynamic Secrets engines can not support a high number of credential requests from replicated workloads. For example, if the Atlas Secrets Engine needed to provision 100 database credentials for 100 pods, this would likely lock any other _vital_ automation in the Atlas environment such as backups or scaling.

The solution to this issue is to run a Vault Agent as a Caching Proxy for credential requests. If all pods use a single k8s service account via the Vault Caching Proxy then the Vault Server only provisions a single instance of the dynamic credential for all 100 pods. The credentials are now "service account scoped" instead of "pod scoped".

**Describe the solution you'd like**

Preferably, the helm chart would support a k8s Deployment that pushes out a cluster (replicated or not) of Vault Agent proxies behind a k8s service.

Currently https://github.com/hashicorp/vault-helm/pull/749 attempts to add the Vault Agent Proxy as a side care for the CSI storage engine. This provides no benefit for the Vault Injector. A standalone proxy would help both and give operators the _control_ they need to confidently administrate Vault workflows.

**Describe alternatives you've considered**

* I've looked at a lot of "operators" that make K8S secrets from Vault, but that introduces a lot of moving parts and we lose the air gapped environment Vault is aiming to provide.

**Additional Context**

**Vault Agent Injector**
* https://github.com/hashicorp/vault-k8s/issues/331
* Wants to be able to configure telemetry on each agent in a pod. Leads to a bunch of low value time series for Prometheus, etc. A central proxy would be easier to configure and provide higher value time series.
* https://github.com/hashicorp/vault-k8s/issues/49
* Vault Agent proxy provides an extra redundancy that can be used on top of a HA Vault.
* HA Vault is great, but is still vulnerable to misconfigurations.

**Secrets CSI Provider**
* https://github.com/hashicorp/vault-csi-provider/issues/150
* CSI Provider doesn't respect auth token TTLs and makes too many requests
* https://github.com/hashicorp/vault-csi-provider/issues/151
* Almost a duplicate of the above, but also indicates issues with secret rotation (on top of auth renewal).
* https://github.com/hashicorp/vault-csi-provider/issues/149
* I believe the issue indicates that in a deployment with 50 pods, 50 credentials are deployed, but the CSI only uses the last credential deployed. Proxy would only provision 1 and the CSI would use that.

Other Technical Advantages

In general, I think there are strong reasons to treat the Vault Agent Proxy as a standalone deployment:

1. HA/DR
+ Deploy multiple instances of a cache with topology aware scheduling to be resilient against zonal failures.
+ Simpler run books: scale up, restart, for an individual component instead of a coupled component.
2. Monitoring
+ Monitoring all Injected Agents for the Vault Injector may be untenable for overloaded prometheus instances.
+ A central cache establishes a good "bottle neck" to monitor the aggregate and then identify the issue.
3. Improve Cache Hit Rates
+ In large clusters it may be valuable to partition Vault Proxies by application to have smaller deployments with higher cache hit rates.
4. More Generic -> More Use Cases
+ Building the Vault Agent proxy into the injector or the CSI is a good idea, but a standalone instance can support more use cases.
+ More use cases means more improvements delivered to a smaller set of files in the code base.

Contributor guide

Open the contributing guide

Research direction

Start with the vault-helm chart and compare the proposed standalone Vault Agent proxy with PR #749's sidecar approach for the CSI storage engine. Define done as a configurable Kubernetes Deployment, optionally replicated, exposed through a Service so both the Injector and CSI workflows can use the central caching proxy.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.