pingcap / pingcap/tidb

The handle allocation of IMPORT INTO FROM SELECT will cause SST overlap

Open
#60,453 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Print SST key ranges

```go
// print SST ranges
ssts, err := engine.db.Load().SSTables()
if err != nil {
log.FromContext(ctx).Error("failed to get SSTables", zap.Error(err))
} else {
for i, sstInLevel := range ssts {
log.FromContext(ctx).Info("sstable in level",
zap.Int("level", i),
zap.Int("sstable count", len(sstInLevel)),
)
for _, sst := range sstInLevel {
log.FromContext(ctx).Info("sstable",
zap.Binary("smallest key", sst.Smallest.UserKey),
zap.Binary("largest key", sst.Largest.UserKey))
}
}
}
```

For this table which does not have explicit handle

```sql
CREATE TABLE `t` (
`id` int NOT NULL,
`k` int NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
`id2` int NOT NULL,
`k2` int NOT NULL DEFAULT '0',
`c2` char(120) NOT NULL DEFAULT '',
`pad2` char(60) NOT NULL DEFAULT '',
KEY `k_1` (`k`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
```

run

```
mysql> import into t from select * from sbtest1 join sbtest2 where sbtest2.id <=2000 with thread=16,DISABLE_PRECHECK,checksum_table='off';
Query OK, 20000000 rows affected (32.37 sec)
Records: 20000000, ID: e1c93031-fb32-459c-a562-50eb71d78bf0
```

we can find these SST information in logs

```
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=0] ["sstable count"=16]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:397] [sstable] [category=ddl-ingest] ["smallest key"="dIAAAAAAAACqX3KAAAAAAAAMAQ=="] ["largest key"="dIAAAAAAAACqX3KAAAAAATERAA=="]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:397] [sstable] [category=ddl-ingest] ["smallest key"="dIAAAAAAAACqX3KAAAAAAAAYAQ=="] ["largest key"="dIAAAAAAAACqX3KAAAAAATEZAA=="]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:397] [sstable] [category=ddl-ingest] ["smallest key"="dIAAAAAAAACqX3KAAAAAAAAgAQ=="] ["largest key"="dIAAAAAAAACqX3KAAAAAATD1AA=="]
...
```

for the 3 example lines of SST, their ranges are

[3073, 19992832]
[6145, 19994880]
[8193, 19985664]

And they are all in level 0. Open an iterator and read the KV will be slow.

other log lines

```
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=1] ["sstable count"=0]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=2] ["sstable count"=0]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=3] ["sstable count"=0]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=4] ["sstable count"=0]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=5] ["sstable count"=0]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:392] ["sstable in level"] [category=ddl-ingest] [level=6] ["sstable count"=1]
[2025/04/09 15:34:56.674 +08:00] [INFO] [engine_mgr.go:397] [sstable] [category=ddl-ingest] ["smallest key"="dIAAAAAAAACqX3KAAAAAAAAAAQ=="] ["largest key"="dIAAAAAAAACqX3KAAAAAATEdAA=="]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.