auth: expose in-memory simple token count
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 44
Description
### What would you like to be added?
Add a Prometheus metric for the number of in-memory simple tokens, and log a warning as the count grows (e.g. every 10,000 tokens).
Simple tokens live in a process-local map and are scanned under lock every second. There is currently no metric or log for the size of that map, so operators cannot see unbounded growth until the member is already unavailable.
### Why is this needed?
Simple tokens are not recommended for production ([auth design](https://etcd.io/docs/latest/learning/design-auth-v3/#two-types-of-tokens-simple-and-jwt)), but they are still the default. In production we saw a cluster become unavailable when:
1. `--auth-token-ttl` was set to `315360000` (10 years), so tokens never expired.
2. Clients called `Authenticate` on every new stream (`LeaseKeepAlive` / Watch) instead of reusing a token.
The in-memory map grew without bound. `simpleTokenTTLKeeper.run` held `simpleTokensMu` while iterating the whole map; `Authenticate` apply takes the same lock, so KV/lease traffic stalled.
Evidence:
- 60s CPU pprof: **98.96%** in `auth.(*simpleTokenTTLKeeper).run` (`mapiternext` / `memhashFallback`)
- Heap: **~2.5GB** in `assignSimpleTokenToUser` / `addSimpleToken`
- `top`: etcd **101% CPU**, **2.3GB RSS**
- Logs: `apply request took too long` (3–5s vs 100ms) on `authenticate`, plus `invalid auth token` as clients retried
I am not proposing to make simple tokens production-ready. Production should still use JWT or mTLS. This is only so operators can see the map growing before the member is pinned.
Contributor guide
Research direction
Locate the simple-token implementation, including simpleTokenTTLKeeper.run and the Authenticate path, and trace where the process-local map is updated and scanned. Add visibility for the in-memory token count and growth warning; done means operators can observe an increasing map before it causes resource or availability problems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- authentication, observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100