pingcap / pingcap/tidb

lightning may panic when meet error

Open
#43,101 1 comment 0 reactions 1 assignee Claimed by @lance6716 View on GitHub
affects-5.3 affects-5.4 affects-6.1 affects-6.5 affects-7.1 affects-7.5 affects-8.1 affects-8.5 component/lightning severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

```
unexpected fault address 0x7f422a02b676
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f422a02b676 pc=0x16d4c16]

goroutine 636 [running]:
runtime.throw({0x45bd9fd?, 0x0?})
/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc252fb5720 sp=0xc252fb56f0 pc=0x169e5bd
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:842 +0x2c5 fp=0xc252fb5770 sp=0xc252fb5720 pc=0x16b5425
runtime.memmove()
/usr/local/go/src/runtime/memmove_amd64.s:190 +0x156 fp=0xc252fb5778 sp=0xc252fb5770 pc=0x16d4c16
github.com/pingcap/tidb/br/pkg/lightning/backend/local.noopKeyAdapter.Encode(...)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/key_adapter.go:50
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*noopKeyAdapter).Encode(0xc0b44f2e00?, {0x7f417a2adec6?, 0xc003b5c780?, 0xc23fef96f0?}, {0x7f422a02b676?, 0x7f418bb3f200?, 0x0?}, 0x0?)
:1 +0x93 fp=0xc252fb57c8 sp=0xc252fb5778 pc=0x3332d33
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Writer).appendRowsUnsorted(0xc0b44f2e80, {0x4d13bc8, 0xc003b5c780}, {0xc07deb2000?, 0x63, 0xc0c8af99b0?})
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/engine.go:1074 +0x319 fp=0xc252fb5920 sp=0xc252fb57c8 pc=0x3302ab9
(stack is missing in log)
```

in all versions, lightning didn't wait the `deliverLoop` goroutine exit

https://github.com/pingcap/tidb/blob/8d16b2656c3abcb36d21d4094694cc43f2981db6/br/pkg/lightning/importer/chunk_process.go#L174-L181

So it may still insert data after the `*membuf.Buffer` is closed. Due to we use a CGO allocator we trigger the use-after-free panic

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.