WebSocket Timeout and Stale Cache Issue in Vault Proxy (v1.16.3)
- Dominant language
- Go
- Stars
- 36.3k
- Forks
- 4.8k
- PR merge metrics
- PR metrics pending
Description
Hi Team,
We're currently using Vault Proxy to cache and auto-authenticate static secrets for our applications. While caching appears to function correctly, we've encountered two issues with version 1.16.3 (both the Proxy Agent and Enterprise Server):
1. WebSocket Connection Timeout
The WebSocket connection consistently times out after 20 minutes, despite the auto-auth token being configured with a 4-hour expiry. We haven’t found any explicit configuration that would cause this timeout.
Error observed:
[ERROR] proxy.cache.staticsecretcacheupdater: error occurred during streaming static secret cache update events: err="error returned when opening event stream web socket to wss://hcvaultdv.us.platforms.dev/v1/sys/events/subscribe/kv%2A?json=true, ensure auto-auth token has correct permissions and Vault is Enterprise version 1.16 or above: failed to WebSocket dial: expected handshake response status code 101 but got 400"
What we've verified:
- Auto-auth token is functioning correctly.
- Token has all required capabilities.
- Requests made during the timeout period are processed successfully without errors.
2. Stale Cache Entries
Possibly related to the above issue, cache entries are not auto-updating via the event notification system. As a result, we’re receiving stale values even though the secrets have been updated in Vault.
Additional observations:
- No error logs are generated before or after the secret update.
- The only error logged is the WebSocket 400 error mentioned above.
What we've tried:
- Verified token capabilities and access.
- Able to fetch updated values when specifying the version explicitly; omitting the version returns stale data.
- prepend_configured_namespace is not set.
- All relevant configurations, including allow_forwarding_header, are in place.
Please let us know if you need any further details or logs to help investigate.
Thanks so much!
Kind regards, Shefali
Contributor guide
Research direction
Start with the proxy.cache.staticsecretcacheupdater component and the WebSocket event-stream request to /v1/sys/events/subscribe/kv%2A?json=true, then reproduce the 20-minute timeout on Vault v1.16.3. Trace why the 400 handshake stops cache event updates; done means the connection remains usable for the configured token lifetime and updated secrets no longer remain stale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100