pingcap / pingcap/tidb-operator

Local reads via Stale Reads do not work

Open
#4,627 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of TiDB Operator are you using?
v1.3.5
TidbCluster kind v6.1.0

What did you do?
I try to perform local reads using Stale Reads described in https://docs.pingcap.com/tidb/stable/three-dc-local-read#perform-local-read-using-stale-read, but TIDB sends commands to all tikv across all zones.

What did you expect to see?
Per documentation it should send requests to TiKV in the same zone

I use the following PD config:

      [replication]
        max-replicas = 3
        location-labels = ["topology.kubernetes.io/zone", "kubernetes.io/hostname"]

I also see that corresponding labels are assigned to TiKV nodes:

[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ab73507c-7tqz\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-b\"]"] [store_id=1]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-ac1d383d-09th\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-c\"]"] [store_id=4]
[2022/07/14 00:45:20.651 +00:00] [INFO] [replication_mode.rs:55] ["associated store labels"] [labels="[key: \"kubernetes.io/hostname\" value: \"gke-non-mesh-tidb-pe-basic-tikv-n2d-8-753c179c-tm45\", key: \"topology.kubernetes.io/zone\" value: \"us-east4-a\"]"] [store_id=5]

However, I don't see same labels assigned to TiDB nodes, hence the feature does not work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the documented stale-read procedure and the supplied PD replication configuration, then reproduce using TiDB Operator v1.3.5 and TiDB v6.1.0. Trace the generated TiDB and TiKV pod metadata to compare topology labels, and verify that the resulting local-read requests stay within the expected zone.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.