etcd-io / etcd-io/etcd

grpcproxy: range cache treats an open-ended range end ("\x00") as its start key, serializable reads through the proxy go stale after writes

Open
#22,397 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### Bug report criteria

- [X] This bug report is not security related, security issues should be disclosed privately via [the report form](https://github.com/etcd-io/etcd/security/advisories/new).
- [X] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [X] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [X] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

Through a grpc-proxy, a serializable `get --from-key` keeps returning its cached result after a put above the start key, and `del --from-key` leaves cached reads above the start key in place. Linearizable reads through the same proxy are correct.

The range cache builds its interval-tree entries from the raw wire RangeEnd (server/proxy/grpcproxy/cache/store.go:97 in Add, :145 in Invalidate). A RangeEnd of a single zero byte, which the server executes as "every key >= key" (mkGteRange in server/etcdserver/txn/delete.go:63, api/etcdserverpb/rpc.proto:561), becomes a StringAffineInterval whose end sorts below its begin, and such an interval only ever intersects its own start key.

Both directions on main at d848b43ae, one member, a grpc-proxy on 127.0.0.1:23790:

```
$ etcdctl --endpoints http://127.0.0.1:23790 put a 1
OK
$ etcdctl --endpoints http://127.0.0.1:23790 put z 1
OK
$ etcdctl --endpoints http://127.0.0.1:23790 get a --from-key --consistency=s --keys-only
a

z

$ etcdctl --endpoints http://127.0.0.1:23790 put b 2
OK
$ etcdctl --endpoints http://127.0.0.1:23790 get a --from-key --consistency=s --keys-only
a

z

$ etcdctl --endpoints http://127.0.0.1:23790 get a --from-key --consistency=l --keys-only
a

b

z

$ etcdctl --endpoints http://127.0.0.1:23790 get z --consistency=s --keys-only
z

$ etcdctl --endpoints http://127.0.0.1:23790 del a --from-key
3
$ etcdctl --endpoints http://127.0.0.1:23790 get z --consistency=s --keys-only
z

$ etcdctl --endpoints http://127.0.0.1:23790 get z --consistency=l --keys-only
```

`WithPrefix()` on an empty key sends the same wire shape (key and range end both "\x00"), so a cached whole-keyspace serializable range behaves the same way. The two lines are unchanged on release-3.6 and release-3.5.

### What did you expect to happen?

The cache treats a single zero byte as an open-ended range end the way watch.go:325 and the auth range cache (server/auth/range_perm_cache.go:185) already do: the second serializable `get --from-key a` returns a, b, z and the last `get z` returns nothing.

### How can we reproduce it (as minimally and precisely as possible)?

`etcd` with defaults, `etcd grpc-proxy start --endpoints http://127.0.0.1:12379 --listen-addr 127.0.0.1:23790` (the range cache is always on), then the commands above against the proxy. A table test on cache.Cache that pins both directions with finite-range controls is in the linked PR; its six open-ended rows fail on main.

### Anything else we need to know?

Same shape as the txn overlap guard fixed in #22395, which pointed at these two sites and left them out. Not security related, auth checks run on the server; this is stale serializable data from the proxy cache.

### Etcd version (please run commands below)

main at d848b43ae (etcd Version: 3.8.0-alpha.0, built with scripts/build.sh); the cache code is the same on release-3.6 and release-3.5.

### Etcd configuration (command line flags or environment variables)

Defaults, single member.

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

N/A

### Relevant log output

N/A

Contributor guide

Open the contributing guide

Research direction

Start with server/grpcproxy/cache/store.go at Add and Invalidate, then compare the open-ended range handling in watch.go and server/auth/range_perm_cache.go. Run the linked Cache table test and reproduce the proxy commands against a single member. Done means serializable reads are invalidated for keys at or above the start key after puts and deletes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.