Connection won't be re-established if DNS has a delay
Open
affects-8.5
component/mcs
severity/major
type/bug
type/cse
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
We have a cache to manage the mapping between URL and gRPC connection:
https://github.com/tikv/pd/blob/e2ce37be70eb478137bd22b34843c5fff8481d28/client/servicediscovery/tso_service_discovery.go#L124
But if the DNS has a delay, the gRPC connection won't be re-established. If the pod IP can be reused, there is a possibility that the request is sent to the wrong place. We have encountered this issue in the serverless environment, which causes the tso request to be sent to the tso server, which is not the primary. And the client keeps trying at the same connection, which is wrong until the TiDB is restarted.
Contributor guide
Assessment
This issue has not been assessed yet.