pingcap / pingcap/tidb

lightning cannot exit when task cancelled and there's pending flush chunks, if we start again might cause "resource temporarily unavailable"

Open
#45,155 0 comments 0 reactions 0 assignees View on GitHub
affects-6.1 affects-6.5 affects-7.1 affects-8.1 affects-8.5 component/lightning may-affects-5.4 may-affects-7.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

- if chunk flush is slow, so there might be many pending flush chunks add in here
https://github.com/pingcap/tidb/blob/ab4c06aac5f8cd3ffbb18ef1f6ac5f25d9d40916/br/pkg/lightning/importer/table_import.go#L779
- if they're flushed later, we will put them into a channel to trigger checkpoint save
https://github.com/pingcap/tidb/blob/ab4c06aac5f8cd3ffbb18ef1f6ac5f25d9d40916/br/pkg/lightning/importer/table_import.go#L694
- if we cancel the task, the routine which handles previous channel will exit, so the channel might be full, and block the exit
https://github.com/pingcap/tidb/blob/ab4c06aac5f8cd3ffbb18ef1f6ac5f25d9d40916/br/pkg/lightning/importer/table_import.go#L658-L667

the stack when it's stuck: [stack2.log](https://github.com/pingcap/tidb/files/11948867/stack2.log)

if we try to start another run of same table, we will meet `resource temporarily unavailable`, since some files in pebble db is locked by the un-exited lightning and we're using tablename to generate pebble db directory name.

### 2. What did you expect to see? (Required)
lightning exited
### 3. What did you see instead (Required)
keeps running
### 4. What is your TiDB version? (Required)
hash 60784a30d041479769aa6a4a351df5eeddd0d94a

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.