pingcap / pingcap/tidb

Stop the SQL listener and delete it from topology path before graceful shutdown

Open
#37,471 0 comments 0 reactions 0 assignees View on GitHub
sig/sql-infra type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Background

Currently, when a TiDB gracefully shutdown:
- It returns `500` error from the HTTP status port, but it doesn't close the listener and still accepts new connections. As the doc says:
> If a load balancer is running (recommended): Stop the load balancer and execute the SQL statement SHUTDOWN. Then TiDB waits for a period as specified by [graceful-wait-before-shutdown](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#graceful-wait-before-shutdown-new-in-v50) until all sessions are terminated. Then TiDB stops running.

- It doesn't remove itself from the `/topology/tidb/ip:port/ttl` or `/topology/tidb/ip:port/info` from PD etcd. These 2 paths are never removed.

## Problems

For session manager, which keeps client connections alive and dynamically balances client connections, faces these problems:
- It has to listen to the HTTP port to know whether the TiDB instance is going to shutdown. However, the HTTP port is unsafe and may be removed in the future. Besides, the HTTP port may hang but the SQL port may still work in some cases.
- There will be more and more tombstone TiDB instances registered in the etcd path. Session manager may waste time and network bandwidth to fetch the topology. And it cannot know that a TiDB instance is about to shutdown immediately. It has to visit the HTTP port periodically.

## Proposal

- Stop the SQL listener so that it doesn't accept new connections.
- Remove itself from the `/topology/tidb/ip:port/ttl` and`/topology/tidb/ip:port/info` from PD etcd.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.