pingcap / pingcap/tidb

txn: tidb returns unknown error during upgrading(pd's leader changes)

Open
#46,390 1 comment 0 reactions 0 assignees View on GitHub
sig/transaction type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
When we try to upgrade cluster from v6.5.2 to v6.5.3, pd transfer is leader in 2 seconds, tidb returns unknown error
```
[2023/08/10 04:04:46.824 +00:00] [INFO] [conn.go:1181] ["command dispatched failed"] [conn=xxx] [connInfo="id:xxx, addr:[xxx:xx](http://xxx:xx/) status:1, collation:xx, user:xxx"] [command=Query] [status="inTxn:1, autocommit:0"] [sql="update ] [txn_mode=PESSIMISTIC] [timestamp=443453311336083641] [err="rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster\[ngithub.com/tikv/pd/client.(*client).processTSORequests\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1097\ngithub.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:847\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ngithub.com/tikv/pd/client.(*tsoRequest).Wait\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1313\ngithub.com/tikv/client-go/v2/util.interceptedTsFuture.Wait\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/util/pd_interceptor.go:73\ngithub.com/tikv/client-go/v2/oracle/oracles.(*tsFuture).Wait\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/oracle/oracles/pd.go:116\ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getOracleFuture.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:164\ngithub.com/pingcap/tidb/sessiontxn/isolation.funcFuture.Wait\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:485\ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getStmtTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:189\ngithub.com/pingcap/tidb/sessiontxn/isolation.(*baseTxnContextProvider).GetStmtForUpdateTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:201\ngithub.com/pingcap/tidb/session.(*txnManager).GetStmtForUpdateTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txnmanager.go:92\ngithub.com/pingcap/tidb/executor.(*executorBuilder).updateForUpdateTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2376\ngithub.com/pingcap/tidb/executor.(*executorBuilder).buildUpdate\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2281\ngithub.com/pingcap/tidb/executor.(*executorBuilder).build\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:249\ngithub.com/pingcap/tidb/executor.(*ExecStmt).buildExecutor\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1093\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:500\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2350\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2214\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:252\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2122\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1972\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1403\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1152\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:648\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ngithub.com/tikv/client-go/v2/oracle/oracles.(*tsFuture).Wait\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/oracle/oracles/pd.go:119\ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getOracleFuture.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:164\ngithub.com/pingcap/tidb/sessiontxn/isolation.funcFuture.Wait\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:485\ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getStmtTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:189\ngithub.com/pingcap/tidb/sessiontxn/isolation.(*baseTxnContextProvider).GetStmtForUpdateTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:201\ngithub.com/pingcap/tidb/session.(*txnManager).GetStmtForUpdateTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txnmanager.go:92\ngithub.com/pingcap/tidb/executor.(*executorBuilder).updateForUpdateTS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2376\ngithub.com/pingcap/tidb/executor.(*executorBuilder).buildUpdate\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2281\ngithub.com/pingcap/tidb/executor.(*executorBuilder).build\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:249\ngithub.com/pingcap/tidb/executor.(*ExecStmt).buildExecutor\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1093\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:500\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2350\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2214\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:252\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2122\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1972\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1403\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1152\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:648\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594](http://ngithub.com/tikv/pd/client.(*client).processTSORequests/n/t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1097/ngithub.com/tikv/pd/client.(*client).handleDispatcher/n/t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:847/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1594/ngithub.com/tikv/pd/client.(*tsoRequest).Wait/n/t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1313/ngithub.com/tikv/client-go/v2/util.interceptedTsFuture.Wait/n/t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/util/pd_interceptor.go:73/ngithub.com/tikv/client-go/v2/oracle/oracles.(*tsFuture).Wait/n/t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/oracle/oracles/pd.go:116/ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getOracleFuture.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:164/ngithub.com/pingcap/tidb/sessiontxn/isolation.funcFuture.Wait/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:485/ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getStmtTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:189/ngithub.com/pingcap/tidb/sessiontxn/isolation.(*baseTxnContextProvider).GetStmtForUpdateTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:201/ngithub.com/pingcap/tidb/session.(*txnManager).GetStmtForUpdateTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txnmanager.go:92/ngithub.com/pingcap/tidb/executor.(*executorBuilder).updateForUpdateTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2376/ngithub.com/pingcap/tidb/executor.(*executorBuilder).buildUpdate/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2281/ngithub.com/pingcap/tidb/executor.(*executorBuilder).build/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:249/ngithub.com/pingcap/tidb/executor.(*ExecStmt).buildExecutor/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1093/ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:500/ngithub.com/pingcap/tidb/session.runStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2350/ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2214/ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:252/ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2122/ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1972/ngithub.com/pingcap/tidb/server.(*clientConn).dispatch/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1403/ngithub.com/pingcap/tidb/server.(*clientConn).Run/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1152/ngithub.com/pingcap/tidb/server.(*Server).onConn/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:648/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1594/ngithub.com/tikv/client-go/v2/oracle/oracles.(*tsFuture).Wait/n/t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/oracle/oracles/pd.go:119/ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getOracleFuture.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:164/ngithub.com/pingcap/tidb/sessiontxn/isolation.funcFuture.Wait/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:485/ngithub.com/pingcap/tidb/sessiontxn/isolation.(*PessimisticRCTxnContextProvider).getStmtTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/readcommitted.go:189/ngithub.com/pingcap/tidb/sessiontxn/isolation.(*baseTxnContextProvider).GetStmtForUpdateTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessiontxn/isolation/base.go:201/ngithub.com/pingcap/tidb/session.(*txnManager).GetStmtForUpdateTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txnmanager.go:92/ngithub.com/pingcap/tidb/executor.(*executorBuilder).updateForUpdateTS/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2376/ngithub.com/pingcap/tidb/executor.(*executorBuilder).buildUpdate/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2281/ngithub.com/pingcap/tidb/executor.(*executorBuilder).build/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:249/ngithub.com/pingcap/tidb/executor.(*ExecStmt).buildExecutor/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1093/ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:500/ngithub.com/pingcap/tidb/session.runStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2350/ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2214/ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:252/ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2122/ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1972/ngithub.com/pingcap/tidb/server.(*clientConn).dispatch/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1403/ngithub.com/pingcap/tidb/server.(*clientConn).Run/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1152/ngithub.com/pingcap/tidb/server.(*Server).onConn/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:648/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1594)"]
```

The error was throwed by the following function L164, while here we could do some retry to get-tso again
https://github.com/pingcap/tidb/blob/29116c0256c52b224da2b34d712c1063d171c0ad/sessiontxn/isolation/readcommitted.go#L160-L176

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.