TiDB nightly panic in mockstore/unistore badger Table.HasOverlap during transaction rollback cleanup
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
I used the official `pingcap/tidb:nightly` Docker image. The container uses TiDB's default local `unistore` storage.
Start TiDB nightly:
~~~bash
sudo docker run -d \
--name tidb-nightly \
-p 14005:4000 \
-p 10085:10080 \
pingcap/tidb:nightly
~~~
Run SQLancer against this TiDB instance:
~~~bash
java -jar target/sqlancer-2.0.0.jar --num-threads 16 --num-tries 1000 --host 127.0.0.1 --port 14005 --username root --password '' tidb --oracle VP --vp-query-variants-per-combination 50 --vp-helper-use-base-engine true --vp-source-copy-schema-objects true
~~~
The workload repeatedly creates and drops tables/views in multiple databases. Around the crash time, many concurrent DDL jobs were running, including CREATE VIEW, DROP VIEW, and DROP TABLE.
Examples of SQLancer statements near the crash:
~~~sql
CREATE VIEW `vp_database0_11439_v` AS
SELECT l.`vp_rowid` AS `vp_rowid`, l.`c0` AS `c0`
FROM `vp_database0_11439_l` l
JOIN `vp_database0_11439_r` r ON l.`vp_rowid` = r.`vp_rowid`;
CREATE VIEW `vp_database10_11446_v` AS
SELECT l.`vp_rowid` AS `vp_rowid`, r.`c0` AS `c0`
FROM `vp_database10_11446_l` l
JOIN `vp_database10_11446_r` r ON l.`vp_rowid` = r.`vp_rowid`;
CREATE VIEW `vp_database16_11451_v` AS
SELECT l.`vp_rowid` AS `vp_rowid`, r.`c0` AS `c0`
FROM `vp_database16_11451_l` l
JOIN `vp_database16_11451_r` r ON l.`vp_rowid` = r.`vp_rowid`;
DROP VIEW IF EXISTS `vp_database18_11440_v`;
~~~
One SQLancer thread ended near the following statements:
~~~sql
DROP VIEW IF EXISTS `vp_database18_11440_v`; -- 1514ms;
CREATE VIEW `vp_database18_11440_v` AS
SELECT l.`vp_rowid` AS `vp_rowid`, r.`c0` AS `c0`
FROM `vp_database18_11440_l` l
JOIN `vp_database18_11440_r` r ON l.`vp_rowid` = r.`vp_rowid`; -- 541ms;
SELECT (((CASE (~ (NULL))
WHEN vp_database18_11440_source.c0 THEN -406625085
ELSE (('')|(vp_database18_11440_source.c0))
END ))AND(vp_database18_11440_source.c0))
FROM `vp_database18_11440_v` `vp_database18_11440_source`
ORDER BY vp_database18_11440_source.vp_rowid;
DROP VIEW IF EXISTS `vp_database18_11440_v`;
~~~
However, the panic did not appear to be caused by one specific SQL statement directly. It happened asynchronously during transaction cleanup/rollback while multiple SQLancer threads were issuing DDL statements.
### 2. What did you expect to see? (Required)
TiDB should not panic or crash.
Even if SQLancer sends invalid SQL or creates a heavy DDL workload, TiDB should return SQL errors or transaction errors to the client instead of crashing the server process.
In particular, transaction cleanup/rollback in mockstore/unistore should not trigger a nil pointer dereference in badger iterator/table code.
### 3. What did you see instead (Required)
The TiDB process panicked and the Docker container exited with code 2.
Container status:
~~~text
Image=pingcap/tidb:nightly
Status=exited
ExitCode=2
StartedAt=2026-06-13T14:02:06.684768918Z
FinishedAt=2026-06-17T01:09:29.629604338Z
~~~
Panic:
~~~text
panic: runtime error: invalid memory address or nil pointer dereference
panic: Unclosed iterator at time of Txn.Discard.
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x4afe70f]
~~~
Important stack trace:
~~~
github.com/pingcap/badger.(*Txn).Discard
/root/go/pkg/mod/github.com/pingcap/badger@v1.5.1-0.20241015064302-38533b6cbf8d/transaction.go:446
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv/dbreader.(*DBReader).Close
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/dbreader/db_reader.go:398
panic: runtime error: invalid memory address or nil pointer dereference
github.com/pingcap/badger/table/sstable.(*Table).HasOverlap
/root/go/pkg/mod/github.com/pingcap/badger@v1.5.1-0.20241015064302-38533b6cbf8d/table/sstable/table.go:577
github.com/pingcap/badger.(*IteratorOptions).OverlapTable
/root/go/pkg/mod/github.com/pingcap/badger@v1.5.1-0.20241015064302-38533b6cbf8d/iterator.go:215
github.com/pingcap/badger.(*IteratorOptions).OverlapTables
/root/go/pkg/mod/github.com/pingcap/badger@v1.5.1-0.20241015064302-38533b6cbf8d/iterator.go:240
github.com/pingcap/badger.(*levelHandler).appendIterators
/root/go/pkg/mod/github.com/pingcap/badger@v1.5.1-0.20241015064302-38533b6cbf8d/level_handler.go:409
github.com/pingcap/badger.(*Txn).NewIterator
/root/go/pkg/mod/github.com/pingcap/badger@v1.5.1-0.20241015064302-38533b6cbf8d/iterator.go:289
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv/dbreader.NewIterator
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/dbreader/db_reader.go:91
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv/dbreader.(*DBReader).GetIter
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/dbreader/db_reader.go:165
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv.(*MVCCStore).checkCommitted
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/mvcc.go:1407
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv.(*MVCCStore).rollbackKeyReadDB
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/mvcc.go:1382
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv.(*MVCCStore).Rollback
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/mvcc.go:1344
github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv.(*Server).KvBatchRollback
/workspace/source/tidb/pkg/store/mockstore/unistore/tikv/server.go:540
github.com/pingcap/tidb/pkg/store/mockstore/unistore.(*RPCClient).SendRequest
/workspace/source/tidb/pkg/store/mockstore/unistore/rpc.go:256
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch
/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20260423085316-fce9638195b3/txnkv/transaction/cleanup.go:81
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanup.func1
/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20260423085316-fce9638195b3/txnkv/transaction/2pc.go:1680
~~~
The direct panic site is:
~~~text
github.com/pingcap/badger/table/sstable.(*Table).HasOverlap
table.go:577
~~~
The TiDB side was doing transaction cleanup/rollback:
~~~
KvBatchRollback
MVCCStore.Rollback
rollbackKeyReadDB
checkCommitted
DBReader.GetIter
badger.Txn.NewIterator
badger.Table.HasOverlap
~~~
### 4. What is your TiDB version? (Required)
The server crashed, so I could not run SELECT tidb_version() after the crash. The following version information is from the TiDB startup log.
~~~text
Release Version: v9.0.0-beta.2.pre-1651-g33ae9e3cb5
Edition: Community
Git Commit Hash: 33ae9e3cb52f4891ff35407ed736107aa34e84b9
Git Branch: HEAD
UTC Build Time: 2026-05-01 06:07:45
GoVersion: go1.25.9
Race Enabled: false
Kernel Type: Classic
~~~
Docker image information:
~~~text
Image: pingcap/tidb:nightly
Image ID: sha256:098f676041c6a05b54d8ee72269ce98c10507dffdee74219cc0157c5b117d4dd
Image Created: 2026-05-01T06:15:13.554697888Z
~~~
TiDB loaded config shows it was using local unistore:
~~~text
store: unistore
path: /tmp/tidb
port: 4000
status-port: 10080
~~~
### Timeline around the crash
Shortly before the panic, badger completed a compaction:
~~~text
[2026/06/17 01:09:15.101 +00:00] [INFO] [levels.go:546] ["compact send discard stats"] [stats="numSkips:0, skippedBytes:0"]
[2026/06/17 01:09:15.103 +00:00] [INFO] [levels.go:628] ["compaction done"]
[def="0 top:[0:4](262330584), bot:[10:86](178277388), skip:46, write_amp:1.68"]
[deleted=34]
[added=74]
[duration=25.262939478s]
[2026/06/17 01:09:15.103 +00:00] [INFO] [levels.go:670] ["compaction done"] [level=0]
~~~
About 13ms later, the panic occurred:
~~~text
[2026/06/17 01:09:15.116 +00:00]
panic: runtime error: invalid memory address or nil pointer dereference
~~~
At the same time, multiple DDL jobs were active:
~~~text
jobID=18432257 Type=drop view
jobID=18432258 Type=drop view
jobID=18432261 Type=drop table
jobID=18432263 Type=drop view
jobID=18432274 Type=create view
jobID=18432276 Type=create view
jobID=18432278 Type=create view
~~~
Example DDL log entries near the panic:
~~~text
[2026/06/17 01:09:14.990 +00:00] [INFO] [executor.go:6860] ["DDL job submitted"]
[job="ID:18432274, Type:create view, ..."]
[query="CREATE VIEW `vp_database10_11446_v` AS SELECT l.`vp_rowid` AS `vp_rowid`, r.`c0` AS `c0` FROM `vp_database10_11446_l` l JOIN `vp_database10_11446_r` r ON l.`vp_rowid` = r.`vp_rowid`;"]
[2026/06/17 01:09:14.990 +00:00] [INFO] [executor.go:6860] ["DDL job submitted"]
[job="ID:18432276, Type:create view, ..."]
[query="CREATE VIEW `vp_database16_11451_v` AS SELECT l.`vp_rowid` AS `vp_rowid`, r.`c0` AS `c0` FROM `vp_database16_11451_l` l JOIN `vp_database16_11451_r` r ON l.`vp_rowid` = r.`vp_rowid`;"]
[2026/06/17 01:09:14.990 +00:00] [INFO] [executor.go:6860] ["DDL job submitted"]
[job="ID:18432278, Type:create view, ..."]
[query="CREATE VIEW `vp_database0_11439_v` AS SELECT l.`vp_rowid` AS `vp_rowid`, l.`c0` AS `c0` FROM `vp_database0_11439_l` l JOIN `vp_database0_11439_r` r ON l.`vp_rowid` = r.`vp_rowid`;"]
~~~
### Possible root cause
This looks like a race or lifecycle bug in the mockstore/unistore + badger path.
The panic happens while constructing a badger iterator during transaction rollback cleanup. The iterator code checks SSTable overlap through sstable.Table.HasOverlap, but one of the table's internal fields appears to be nil or invalid.
The suspicious ordering is:
- SQLancer generates many concurrent DDL operations.
- TiDB/unistorre creates many MVCC writes/deletes and async 2PC cleanup tasks.
- badger completes a compaction.
- Shortly after compaction, transaction rollback cleanup calls badger.Txn.NewIterator.
- sstable.Table.HasOverlap panics with nil pointer dereference.
- Txn.Discard also panics because an iterator was left unclosed during the panic path.
This may be related to SSTable lifecycle or iterator construction racing with compaction in badger under TiDB unistore.
Contributor guide
Assessment
This issue has not been assessed yet.