etcd-io / etcd-io/etcd

auth: expose in-memory simple token count

Open
#22,337 0 comments 0 reactions 0 assignees View on GitHub
type/feature
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

### What would you like to be added?

Add a Prometheus metric for the number of in-memory simple tokens, and log a warning as the count grows (e.g. every 10,000 tokens).

Simple tokens live in a process-local map and are scanned under lock every second. There is currently no metric or log for the size of that map, so operators cannot see unbounded growth until the member is already unavailable.

### Why is this needed?

Simple tokens are not recommended for production ([auth design](https://etcd.io/docs/latest/learning/design-auth-v3/#two-types-of-tokens-simple-and-jwt)), but they are still the default. In production we saw a cluster become unavailable when:

1. `--auth-token-ttl` was set to `315360000` (10 years), so tokens never expired.
2. Clients called `Authenticate` on every new stream (`LeaseKeepAlive` / Watch) instead of reusing a token.

The in-memory map grew without bound. `simpleTokenTTLKeeper.run` held `simpleTokensMu` while iterating the whole map; `Authenticate` apply takes the same lock, so KV/lease traffic stalled.

Evidence:
- 60s CPU pprof: **98.96%** in `auth.(*simpleTokenTTLKeeper).run` (`mapiternext` / `memhashFallback`)
- Heap: **~2.5GB** in `assignSimpleTokenToUser` / `addSimpleToken`
- `top`: etcd **101% CPU**, **2.3GB RSS**
- Logs: `apply request took too long` (3–5s vs 100ms) on `authenticate`, plus `invalid auth token` as clients retried
Image

Image

Image

Image

I am not proposing to make simple tokens production-ready. Production should still use JWT or mTLS. This is only so operators can see the map growing before the member is pinned.

Contributor guide

Open the contributing guide

Research direction

Locate the simple-token implementation, including simpleTokenTTLKeeper.run and the Authenticate path, and trace where the process-local map is updated and scanned. Add visibility for the in-memory token count and growth warning; done means operators can observe an increasing map before it causes resource or availability problems.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
authentication, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.