pingcap / pingcap/tidb-operator
Local reads via Stale Reads do not work
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Bug Report
What version of TiDB Operator are you using?
v1.3.5
TidbCluster kind v6.1.0
What did you do?
I try to perform local reads using Stale Reads described in https://docs.pingcap.com/tidb/stable/three-dc-local-read#perform-local-read-using-stale-read, but TIDB sends commands to all tikv across all zones.
What did you expect to see?
Per documentation it should send requests to TiKV in the same zone
I use the following PD config:
[replication]
max-replicas = 3
location-labels = ["topology.kubernetes.io/zone", "kubernetes.io/hostname"]
I also see that corresponding labels are assigned to TiKV nodes:
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ab73507c-7tqz\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-b\"]"] [store_id=1]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ac1d383d-09th\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-c\"]"] [store_id=4]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-753c179c-tm45\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-a\"]"] [store_id=5]
However, I don't see same labels assigned to TiDB nodes, hence the feature does not work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the documented stale-read procedure and the supplied PD replication configuration, then reproduce using TiDB Operator v1.3.5 and TiDB v6.1.0. Trace the generated TiDB and TiKV pod metadata to compare topology labels, and verify that the resulting local-read requests stay within the expected zone.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100