tikv / tikv/pd

tidb abnormally restart with panic when change pd ms mode

Open
#10,329 2 comments 0 reactions 1 assignee Claimed by @JmPotato View on GitHub
affects-8.5 severity/major type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
1、change pd ms mode

### What did you expect to see?
tidb are normal

### What did you see instead?
tidb abnormally restart with panic
[2026/03/11 07:18:15.190 +08:00] [PANIC] [dispatcher.go:519] ["[tso] timestamp fallback"] [keyspace=4294967295] [last-ts="(1773184695175, 4)"] [cur-ts="(1773184695149, 1)"] [last-tso-server=tc-pd-0.tc-pd-peer.pdms-oltp-tpcc-ops-tps-8081031-1-371.svc.cluster.local:2379] [cur-tso-server=tc-pd-2.tc-pd-peer.pdms-oltp-tpcc-ops-tps-8081031-1-371.svc.cluster.local:2379] [last-keyspace-group-in-request=0] [cur-keyspace-group-in-request=0] [last-keyspace-group-in-response=0] [cur-keyspace-group-in-response=0] [last-response-received-at=2026/03/11 07:18:15.177 +08:00] [cur-response-received-at=2026/03/11 07:18:15.190 +08:00] [last-stream-id=tc-pd-0.tc-pd-peer.pdms-oltp-tpcc-ops-tps-8081031-1-371.svc.cluster.local:2379-12] [cur-stream-id=tc-pd-2.tc-pd-peer.pdms-oltp-tpcc-ops-tps-8081031-1-371.svc.cluster.local:2379-14]
[2026/03/11 07:18:15.190 +08:00] [FATAL] [stream.go:330] ["tsoStream.recvLoop internal panic"] [stacktrace="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/stream.go:330\nruntime.gopanic\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.6.linux-amd64/src/runtime/panic.go:783\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.1/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.1/zapcore/entry.go:272\ngo.uber.org/zap.(*Logger).Panic\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.1/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20250917021125-19901e015dc9/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/dispatcher.go:519\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/dispatcher.go:459\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/stream.go:444"] [panicMessage="[tso] timestamp fallback"] [stack="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/stream.go:330\nruntime.gopanic\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.6.linux-amd64/src/runtime/panic.go:783\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.1/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.1/zapcore/entry.go:272\ngo.uber.org/zap.(*Logger).Panic\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.1/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20250917021125-19901e015dc9/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/dispatcher.go:519\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/dispatcher.go:459\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20260122075414-848dd85011e9/clients/tso/stream.go:444"]

### What version of PD are you using (`pd-server -V`)?
./pd-server -V
Release Version: v9.0.0-beta.2.pre-311-ga7dbbf8
Edition: Community
Kernel Type: Classic
Git Commit Hash: a7dbbf8aba92d8ddedd6ef6e070b28ef0671e40f
Git Branch: HEAD
UTC Build Time: 2026-03-10 07:15:02
2026-03-11T06:29:20.914+0800

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.