Panic during scale out

What did you do?

Perform a scale out action

What did you expect to see?

No response

What did you see instead?

[2025/11/25 21:37:53.831 +00:00] [PANIC] [maintainer_controller_bootstrap.go:381] ["map is out of order"] [lastSpan="TableID:717 StartKey:\"x\\000\\000\\001t\\200\\000\\000\\377\\000\\000\\000\\002\\315_r\\230\\377\\000\\000\\246\\301B\\302\\242\\000\\376\" EndKey:\"x\\000\\000\\001t\\200\\000\\000\\377\\000\\000\\000\\002\\315_r\\250\\377\\000\\000\\245c\\275t\\250\\000\\376\" KeyspaceID:1 "] [current="TableID:717 StartKey:\"x\\000\\000\\001t\\200\\000\\000\\377\\000\\000\\000\\002\\315_r\\230\\377\\000\\000\\246\\367K\\213\\014\\000\\376\" EndKey:\"x\\000\\000\\001t\\200\\000\\000\\377\\000\\000\\000\\002\\315_r\\240\\377\\000\\000\\243\\235X\\330X\\000\\376\" KeyspaceID:1 "] [stack="github.com/pingcap/ticdc/maintainer.findHoles.func1\n\tgithub.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:381\ngithub.com/pingcap/ticdc/utils.(*BtreeMap[...]).Ascend.func1\n\tgithub.com/pingcap/ticdc/utils/btree_map.go:98\ngithub.com/google/btree.(*node[...]).iterate\n\tgithub.com/google/btree@v1.1.2/btree_generic.go:522\ngithub.com/google/btree.(*BTreeG[...]).Ascend\n\tgithub.com/google/btree@v1.1.2/btree_generic.go:779\ngithub.com/pingcap/ticdc/utils.(*BtreeMap[...]).Ascend\n\tgithub.com/pingcap/ticdc/utils/btree_map.go:97\ngithub.com/pingcap/ticdc/maintainer.findHoles\n\tgithub.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:370\ngithub.com/pingcap/ticdc/maintainer.(*Controller).handleTableHoles\n\tgithub.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:242\ngithub.com/pingcap/ticdc/maintainer.(*Controller).processTableSpans\n\tgithub.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:227\ngithub.com/pingcap/ticdc/maintainer.(*Controller).processTablesAndBuildSchemaInfo\n\tgithub.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:195\ngithub.com/pingcap/ticdc/maintainer.(*Controller).FinishBootstrap\n\tgithub.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:103\ngithub.com/pingcap/ticdc/maintainer.(*Maintainer).onBootstrapDone\n\tgithub.com/pingcap/ticdc/maintainer/maintainer.go:903\ngithub.com/pingcap/ticdc/maintainer.(*Maintainer).onMaintainerBootstrapResponse\n\tgithub.com/pingcap/ticdc/maintainer/maintainer.go:841\ngithub.com/pingcap/ticdc/maintainer.(*Maintainer).onMessage\n\tgithub.com/pingcap/ticdc/maintainer/maintainer.go:456\ngithub.com/pingcap/ticdc/maintainer.(*Maintainer).HandleEvent\n\tgithub.com/pingcap/ticdc/maintainer/maintainer.go:324\ngithub.com/pingcap/ticdc/maintainer.(*Maintainer).runHandleEvents\n\tgithub.com/pingcap/ticdc/maintainer/maintainer.go:1128"]
[2025/11/25 21:37:53.831 +00:00] [INFO] [maintainer.go:291] ["maintainer is too slow"] [changefeed=10490128323027520380/changefeed-1993214630348349440] [eventType=1] [duration=1.34697017s] [Message="From: cdbbb572-0f56-4e02-af92-50dcad75941e, To: cdbbb572-0f56-4e02-af92-50dcad75941e, Type: MaintainerBootstrapResponse, Message: [changefeedID:<high:8508566247525056151 low:2903642990129064253 name:\"changefeed-1993214630348349440\" keyspace:\"10490128323027520380\" > spans:<ID:<high:17631200909094435466 low:13638332915870219864 > span:<StartKey:\"x\\000\\000\\001t\\200\\000\\000\\377\\000\\000\\000\\000\\000_r\\000\\376\" EndKey:\"x\\000\\000\\001t\\200\\000\\000\\377\\000\\000\\000\\000\\000_s\\000\\376\" KeyspaceID:1 > component_status:Initializing checkpoint_ts:462449070379893561 > checkpoint_ts:462449070379893561 ]"]
panic: map is out of order

goroutine 3042 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x80?, 0x2?, {0x2?, 0x0?, 0x0?})
	go.uber.org/zap@v1.27.0/zapcore/entry.go:196 +0x78
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x403a7561a0, {0x4050c75280, 0x2, 0x2})
	go.uber.org/zap@v1.27.0/zapcore/entry.go:262 +0x1b8
go.uber.org/zap.(*Logger).Panic(0x404d473378?, {0x56500e7?, 0x404d473378?}, {0x4050c75280, 0x2, 0x2})
	go.uber.org/zap@v1.27.0/logger.go:285 +0x4c
github.com/pingcap/log.Panic({0x56500e7?, 0x62849b0?}, {0x4050c75280?, 0x0?, 0x53bf6e0?})
	github.com/pingcap/log@v1.1.1-0.20250917021125-19901e015dc9/global.go:54 +0x94
github.com/pingcap/ticdc/maintainer.findHoles.func1(0x40447483c0, 0x405100f660?)
	github.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:381 +0x1bc
github.com/pingcap/ticdc/utils.(*BtreeMap[...]).Ascend.func1()
	github.com/pingcap/ticdc/utils/btree_map.go:98 +0x2c
github.com/google/btree.(*node[...]).iterate(0x6386480, 0x44d473588, {{0x0?, 0x0?}, 0x20?}, {{0x0?, 0x0?}, 0xc8?}, 0x0?, 0x0, ...)
	github.com/google/btree@v1.1.2/btree_generic.go:522 +0x308
github.com/google/btree.(*BTreeG[...]).Ascend(0x3e09294?, 0x20?)
	github.com/google/btree@v1.1.2/btree_generic.go:779 +0x50
github.com/pingcap/ticdc/utils.(*BtreeMap[...]).Ascend(...)
	github.com/pingcap/ticdc/utils/btree_map.go:97
github.com/pingcap/ticdc/maintainer.findHoles({0x62b2b88, 0x40508baba0}, 0x405101a180)
	github.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:370 +0x158
github.com/pingcap/ticdc/maintainer.(*Controller).handleTableHoles(0x40447367e0, 0x403a347ec0, {0x4050c75200?, 0x2766c?, 0x60?, 0x4050cfe900?}, {0x62b2b88?, 0x40508baba0?}, 0x404d473748?)
	github.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:242 +0x40
github.com/pingcap/ticdc/maintainer.(*Controller).processTableSpans(0x40447367e0, {0x404d4b7470?, 0x20f29?, 0x60?, 0x4050cfe900?}, 0x4050ebe660, 0x0)
	github.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:227 +0x308
github.com/pingcap/ticdc/maintainer.(*Controller).processTablesAndBuildSchemaInfo(0x40447367e0, {0x4050df7608, 0x2b, 0x275d4?}, 0x4050ebe660, 0x4050ebe690, 0x0)
	github.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:195 +0x64
github.com/pingcap/ticdc/maintainer.(*Controller).FinishBootstrap(0x40447367e0, 0x403b30a060, 0x0)
	github.com/pingcap/ticdc/maintainer/maintainer_controller_bootstrap.go:103 +0x334
github.com/pingcap/ticdc/maintainer.(*Maintainer).onBootstrapDone(0x403a36cb48, 0x403b30a060)
	github.com/pingcap/ticdc/maintainer/maintainer.go:903 +0x6c
github.com/pingcap/ticdc/maintainer.(*Maintainer).onMaintainerBootstrapResponse(0x403a36cb48, 0x40017c75e0)
	github.com/pingcap/ticdc/maintainer/maintainer.go:841 +0x33c
github.com/pingcap/ticdc/maintainer.(*Maintainer).onMessage(0x403a36cb48?, 0x66d24?)
	github.com/pingcap/ticdc/maintainer/maintainer.go:456 +0xd8
github.com/pingcap/ticdc/maintainer.(*Maintainer).HandleEvent(0x403a36cb48, 0x404817eac0)
	github.com/pingcap/ticdc/maintainer/maintainer.go:324 +0x2ac
github.com/pingcap/ticdc/maintainer.(*Maintainer).runHandleEvents(0x403a36cb48, {0x629f788, 0x403bed6870})
	github.com/pingcap/ticdc/maintainer/maintainer.go:1128 +0xe0
created by github.com/pingcap/ticdc/maintainer.NewMaintainer in goroutine 1769
	github.com/pingcap/ticdc/maintainer/maintainer.go:247 +0xef0

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

Start at maintainer/maintainer_controller_bootstrap.go:370-381 in findHoles, then inspect utils/btree_map.go:97-98 and the reported call path through handleTableHoles. Reproduce a scale-out action and compare the logged lastSpan and current ordering. Done means the scale-out path no longer panics with “map is out of order”; the issue provides no test name or version data to guide validation.