hashicorp / hashicorp/vault

Consumption-billing KV/mount counting hammers the S3 storage backend with recursive LIST calls

Open
#32,090 0 comments 1 reaction 0 assignees View on GitHub
bug core/metric storage/s3
Dominant language
Go
Stars
36.3k
Forks
4.8k
PR merge metrics
PR metrics pending

Description

**Describe the bug**

On Vault 2.0.x with the **S3 storage backend**, the consumption-billing metrics collector runs every ~10 minutes and counts KV secrets / enumerates mounts by recursively LISTing the storage backend. On S3 each List is a `ListObjectsV2`, so a KV-v2 tree produces tens of thousands of LIST calls per cycle. Upgrading from 1.21.4 → 2.0.x increased S3 request volume by roughly an order of magnitude, scaling with the number of secrets.

This looks like the same class of issue fixed for LDAP roles in 2.0.4 (moved from LIST-based counting to dedicated count endpoints); the KV/secret-mount counting still uses the LIST-based path, and is still present in 2.0.4.

**To Reproduce**
1. Run Vault 2.0.x with the S3 storage backend and a KV-v2 mount with many secrets.
2. Tail the server logs — every ~10 min: `core: updated local max kv counts`, `… role and managed key counts`, etc.
3. Watch S3 metrics / CloudTrail for the Vault bucket: a large recurring volume of `ListObjectsV2` under `logical//metadata/**` and `auth//secret_id`, on that cadence.

**Expected behavior**
Billing/usage counts shouldn't require recursively LISTing the whole storage backend every cycle — as with the LDAP role fix, KV/secret-mount counts should use count endpoints / incremental counters. On object-store backends this generates continuous request volume and cost proportional to secret count, with no way to disable collection (only `sys/billing/config` retention exists).

**Environment:**
* Vault Server Version: 2.0.3 (also confirmed present in 2.0.4)
* Vault CLI Version: 2.0.3
* Server Operating System/Architecture: Linux / x86_64 (Amazon Linux 2023, EKS)

Vault server configuration file(s):

```hcl
storage "s3" {
bucket = ""
region = ""
}
```

**Additional context**
n/a

Contributor guide

Open the contributing guide

Research direction

Start at the consumption-billing collector behind the `core: updated local max kv counts` log and `sys/billing/config`; compare its KV and secret-mount counting path with the LDAP-role fix in Vault 2.0.4. Reproduce against an S3 backend while watching `ListObjectsV2` calls, then verify that recurring counts no longer recursively list the KV metadata and secret_id trees.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.