Attach backend connection ID in the log

Development Task

Sometimes it's hard to troubleshoot without the backend connection ID.

E.g. when TiProxy prints

[2023/06/12 22:43:19.576 +00:00] [INFO] [main.proxy.conn] [client/client_conn.go:72] [fails to relay the connection] [connID=77] [client_addr=172.30.5.201:44527] [proxy-protocol=false] [backend_addr=db-tidb-1.db-tidb-peer.tidb1379661944646167024.svc:4000] ["quit source"="backend quit"] [error="this is an error from client: failed to read the connection: read tcp 172.30.5.9:6000->172.30.5.201:44527: i/o timeout"] [errorVerbose="this is an error from client: failed to read the connection: read tcp 172.30.5.9:6000->172.30.5.201:44527: i/o timeout\ngithub.com/pingcap/TiProxy/pkg/proxy/net.(*PacketIO).wrapErr\n\t/proxy/pkg/proxy/net/packetio.go:115\ngithub.com/pingcap/TiProxy/pkg/proxy/net.(*PacketIO).ReadPacket\n\t/proxy/pkg/proxy/net/packetio.go:199\ngithub.com/pingcap/TiProxy/pkg/proxy/client.(*ClientConnection).processMsg\n\t/proxy/pkg/proxy/client/client_conn.go:79\ngithub.com/pingcap/TiProxy/pkg/proxy/client.(*ClientConnection).Run\n\t/proxy/pkg/proxy/client/client_conn.go:62\ngithub.com/pingcap/TiProxy/pkg/proxy.(*SQLServer).onConn\n\t/proxy/pkg/proxy/proxy.go:190\ngithub.com/pingcap/TiProxy/pkg/proxy.(*SQLServer).Run.func2.1\n\t/proxy/pkg/proxy/proxy.go:132\ngithub.com/pingcap/TiProxy/lib/util/waitgroup.(*WaitGroup).Run.func1\n\t/proxy/lib/util/waitgroup/waitgroup.go:33\nruntime.goexit\n\t/usr/lib/go/src/runtime/asm_arm64.s:1172"]

It's hard to find why TiDB closes the connection, until I find this log in TiDB:

[2023/06/12 22:42:23.064 +00:00] [INFO] [conn.go:1124] ["read packet timeout, close this connection"] [conn=8770920803001697179] [idle=8h0m0.000026656s] [waitTimeout=28800] [error="read tcp 172.30.4.121:4000->172.30.5.9:60016: i/o timeout"]

If we have the backend connection ID in the log, we can quickly find the TiDB log through the backend connection ID.

Since TiDB connection is 8 bytes but it returns 4 bytes to TiProxy in the handshake, it's not easy to figure out the real backend connection ID. I think it's fine to query TiDB with select connection_id() once the handshake succeeds.

An alternative is to log the local address(especially the port) to the backend connection, because TiDB also prints the TiProxy address.

Start with the backend connection logging in client/client_conn.go, including the log location shown at line 72, and trace the handshake path that could obtain or associate the TiDB connection ID. The work is done when backend connection logs contain an ID that can be matched reliably to the corresponding TiDB log; the issue also identifies local-address logging as an alternative to evaluate.