Consumption-billing KV/mount counting hammers the S3 storage backend with recursive LIST calls
- Dominant language
- Go
- Stars
- 36.3k
- Forks
- 4.8k
- PR merge metrics
- PR metrics pending
Description
**Describe the bug**
On Vault 2.0.x with the **S3 storage backend**, the consumption-billing metrics collector runs every ~10 minutes and counts KV secrets / enumerates mounts by recursively LISTing the storage backend. On S3 each List is a `ListObjectsV2`, so a KV-v2 tree produces tens of thousands of LIST calls per cycle. Upgrading from 1.21.4 → 2.0.x increased S3 request volume by roughly an order of magnitude, scaling with the number of secrets.
This looks like the same class of issue fixed for LDAP roles in 2.0.4 (moved from LIST-based counting to dedicated count endpoints); the KV/secret-mount counting still uses the LIST-based path, and is still present in 2.0.4.
**To Reproduce**
1. Run Vault 2.0.x with the S3 storage backend and a KV-v2 mount with many secrets.
2. Tail the server logs — every ~10 min: `core: updated local max kv counts`, `… role and managed key counts`, etc.
3. Watch S3 metrics / CloudTrail for the Vault bucket: a large recurring volume of `ListObjectsV2` under `logical//metadata/**` and `auth//secret_id`, on that cadence.
**Expected behavior**
Billing/usage counts shouldn't require recursively LISTing the whole storage backend every cycle — as with the LDAP role fix, KV/secret-mount counts should use count endpoints / incremental counters. On object-store backends this generates continuous request volume and cost proportional to secret count, with no way to disable collection (only `sys/billing/config` retention exists).
**Environment:**
* Vault Server Version: 2.0.3 (also confirmed present in 2.0.4)
* Vault CLI Version: 2.0.3
* Server Operating System/Architecture: Linux / x86_64 (Amazon Linux 2023, EKS)
Vault server configuration file(s):
```hcl
storage "s3" {
bucket = ""
region = ""
}
```
**Additional context**
n/a
Contributor guide
Research direction
Start at the consumption-billing collector behind the `core: updated local max kv counts` log and `sys/billing/config`; compare its KV and secret-mount counting path with the LDAP-role fix in Vault 2.0.4. Reproduce against an S3 backend while watching `ListObjectsV2` calls, then verify that recurring counts no longer recursively list the KV metadata and secret_id trees.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100