pingcap / pingcap/tidb-operator
tidb Service resource is constantly being updated
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Bug Report
What version of Kubernetes are you using?
Server Version: v1.31.5
What version of TiDB Operator are you using?
1.6.1
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
zfs-localpv-32k
What's the status of the TiDB cluster pods?
peter@pad:~$ kubectl -n tidb-dev get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
backup-daily-2025-03-10t02-15-00-9kfnz 0/1 Completed 0 17h 10.160.220.198 k8s-wk3.marand.si <none> <none>
dev-discovery-96f7bb6c-k657c 1/1 Running 1 32d 10.160.220.33 k8s-wk2.marand.si <none> <none>
dev-ng-monitoring-0 1/1 Running 0 5d6h 10.160.221.61 k8s-wk3.marand.si <none> <none>
dev-pd-0 1/1 Running 0 5d6h 10.160.200.206 k8s-wk1.marand.si <none> <none>
dev-pd-1 1/1 Running 1 (5d6h ago) 5d6h 10.160.219.206 k8s-wk2.marand.si <none> <none>
dev-pd-2 1/1 Running 0 5d6h 10.160.221.0 k8s-wk3.marand.si <none> <none>
dev-tidb-0 2/2 Running 0 5d6h 10.160.219.235 k8s-wk2.marand.si <none> <none>
dev-tidb-1 2/2 Running 0 5d6h 10.160.220.195 k8s-wk3.marand.si <none> <none>
dev-tidb-2 2/2 Running 0 5d6h 10.160.200.198 k8s-wk1.marand.si <none> <none>
dev-tiflash-0 4/4 Running 0 5d6h 10.160.219.227 k8s-wk2.marand.si <none> <none>
dev-tiflash-1 4/4 Running 0 5d6h 10.160.200.251 k8s-wk1.marand.si <none> <none>
dev-tiflash-2 4/4 Running 0 5d6h 10.160.220.235 k8s-wk3.marand.si <none> <none>
dev-tikv-0 1/1 Running 0 5d6h 10.160.220.205 k8s-wk3.marand.si <none> <none>
dev-tikv-1 1/1 Running 0 5d6h 10.160.200.234 k8s-wk1.marand.si <none> <none>
dev-tikv-2 1/1 Running 0 5d6h 10.160.219.198 k8s-wk2.marand.si <none> <none>
What did you do?
I'm developing a k8s controller (operator) that listens to changes in k8s Service resources. I noticed that of all the Service(s) in my k8s cluster (~ 200) only the 2 Services that route connections to tidb pods of the 2 TiDB clusters are constantly being updated, so reconciliation loop in my controller is constantly being triggered by the changes in those two Service(s). Here's the debug log from my controller:
2025-03-10 20:20:30,167 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:20:30,173 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:20:59,347 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:20:59,353 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:21:00,347 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:21:00,351 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:21:29,547 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:21:29,552 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:21:30,547 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:21:30,553 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:21:59,748 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:21:59,754 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:22:00,349 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:22:00,354 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:22:29,349 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:22:29,357 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-infra/infra-tidb} -> DnsZone{}
2025-03-10 20:22:30,349 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-2) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
2025-03-10 20:22:30,356 DEBUG [si.pel.nod.ope.DnsZoneReconciler] (-2048888303-pool-8-thread-4) secondaryToPrimaryMapper: Service{tidb-dev/dev-tidb} -> DnsZone{}
Checking with kubectl, nothing changes in these 2 Service resources, but the metatata.resourceVersion attribute:
peter@pad:~$ date; kubectl -n tidb-infra get Service infra-tidb -o yaml | yq .metadata.resourceVersion
Mon Mar 10 09:24:42 PM CET 2025
252137056
peter@pad:~$ date; kubectl -n tidb-infra get Service infra-tidb -o yaml | yq .metadata.resourceVersion
Mon Mar 10 09:24:59 PM CET 2025
252137256
and:
peter@pad:~$ kubectl -n tidb-infra get Service infra-tidb -o yaml > tidb-infra-service.1.yaml
peter@pad:~$ kubectl -n tidb-infra get Service infra-tidb -o yaml > tidb-infra-service.2.yaml
peter@pad:~$ diff -c tidb-infra-service.1.yaml tidb-infra-service.2.yaml
*** tidb-infra-service.1.yaml 2025-03-10 21:26:25.892254353 +0100
--- tidb-infra-service.2.yaml 2025-03-10 21:26:56.222241483 +0100
***************
*** 20,26 ****
kind: TidbCluster
name: infra
uid: dda0e131-56fb-41fb-909f-86f99a676e7f
! resourceVersion: "252137633"
uid: 82fd6c12-064a-4dbb-9e07-d875de89ff02
spec:
allocateLoadBalancerNodePorts: true
--- 20,26 ----
kind: TidbCluster
name: infra
uid: dda0e131-56fb-41fb-909f-86f99a676e7f
! resourceVersion: "252137817"
uid: 82fd6c12-064a-4dbb-9e07-d875de89ff02
spec:
allocateLoadBalancerNodePorts: true
Other Service(s) managed by TiDB operator don't exhibit this behavior. Other services in the entire k8s cluster don't exhibiti this behavior. So is perhaps TiDB operator unnecessarily updating the tidb Service resource? This should be optimized.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by reproducing the repeated Kubernetes Service updates with kubectl, then trace the TiDB Operator reconciliation that manages the TiDB Service. Done means identifying and preventing unnecessary updates, with a regression test showing that an unchanged Service does not receive new resource versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100