pingcap / pingcap/tidb

owner manager hang in BreakCampaignLoop

Open
#58,620 1 comment 0 reactions 1 assignee Claimed by @tiancaiamao View on GitHub
affects-8.5 component/ddl severity/minor type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

tikv exit first, and then tidb fail to exit when I press Ctrl+C

### 2. What did you expect to see? (Required)

tidb should not hang when exit.

### 3. What did you see instead (Required)

It hang here:

```
0 0x00000000020b972e in runtime.gopark
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:425
1 0x0000000002094ac5 in runtime.goparkunlock
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430
2 0x0000000002094ac5 in runtime.semacquire1
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/sema.go:178
3 0x00000000020bb165 in sync.runtime_Semacquire
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/sema.go:71
4 0x00000000020d5c68 in sync.(*WaitGroup).Wait
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/sync/waitgroup.go:118
5 0x0000000003c82ab0 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).BreakCampaignLoop
at ./pkg/owner/manager.go:349
6 0x0000000004a6dcc4 in github.com/pingcap/tidb/pkg/ddl.(*ddl).close
at ./pkg/ddl/ddl.go:1032
7 0x0000000004a6b6d7 in github.com/pingcap/tidb/pkg/ddl.(*ddl).Stop
at ./pkg/ddl/ddl.go:765
8 0x0000000004d0df03 in github.com/pingcap/tidb/pkg/domain.(*Domain).Close
at ./pkg/domain/domain.go:1248
9 0x0000000005a461d0 in main.closeDDLOwnerMgrDomainAndStorage
at ./cmd/tidb-server/main.go:904
10 0x0000000005a462de in main.cleanup
at ./cmd/tidb-server/main.go:929
11 0x0000000005a421aa in main.main.func1
at ./cmd/tidb-server/main.go:330
12 0x0000000005a2f0a7 in github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler.func2
at ./pkg/util/signal/signal_posix.go:55
13 0x00000000020c2721 in runtime.goexit
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700
```

ddl close call `d.ownerManager.BreakCampaignLoop()` and the owner manager hang.
It's waiting for the campaign loop to exit, but the campaign loop never break:

```
0 0x00000000020b972e in runtime.gopark
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:425
1 0x00000000020938a5 in runtime.selectgo
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/select.go:335
2 0x0000000002743cd8 in google.golang.org/grpc.(*pickerWrapper).pick
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/picker_wrapper.go:114
3 0x000000000273e64f in google.golang.org/grpc.(*ClientConn).getTransport
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/clientconn.go:1047
4 0x00000000027596c5 in google.golang.org/grpc.(*csAttempt).getTransport
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/stream.go:467
5 0x0000000002758ca5 in google.golang.org/grpc.newClientStreamWithParams.func2
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/stream.go:350
6 0x000000000275a97a in google.golang.org/grpc.(*clientStream).withRetry
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/stream.go:777
7 0x0000000002758768 in google.golang.org/grpc.newClientStreamWithParams
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/stream.go:362
8 0x0000000002757b27 in google.golang.org/grpc.newClientStream.func3
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/stream.go:219
9 0x00000000027574f9 in google.golang.org/grpc.newClientStream
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/stream.go:254
10 0x0000000002738d77 in google.golang.org/grpc.invoke
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/call.go:66
11 0x0000000002da2a98 in go.etcd.io/etcd/client/v3.(*Client).unaryClientInterceptor.func1
at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.12/retry_interceptor.go:58
12 0x0000000002738c45 in google.golang.org/grpc.(*ClientConn).Invoke
at /home/genius/go/pkg/mod/google.golang.org/grpc@v1.63.2/call.go:35
13 0x0000000002d2ef65 in go.etcd.io/etcd/api/v3/etcdserverpb.(*leaseClient).LeaseGrant
at /home/genius/go/pkg/mod/go.etcd.io/etcd/api/v3@v3.5.12/etcdserverpb/rpc.pb.go:6812
14 0x0000000002da03d5 in go.etcd.io/etcd/client/v3.(*retryLeaseClient).LeaseGrant
at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.12/retry.go:144
15 0x0000000002d99452 in go.etcd.io/etcd/client/v3.(*lessor).Grant
at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.12/lease.go:216
16 0x0000000003273c73 in go.etcd.io/etcd/client/v3/concurrency.NewSession
at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.12/concurrency/session.go:46
17 0x000000000327e565 in github.com/pingcap/tidb/pkg/util.NewSession
at ./pkg/util/etcd.go:72
18 0x0000000003c83c65 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).refreshSession
at ./pkg/owner/manager.go:446
19 0x0000000003c82cbb in github.com/pingcap/tidb/pkg/owner.(*ownerManager).campaignLoop
at ./pkg/owner/manager.go:367
20 0x0000000003c827c8 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).CampaignOwner.gowrap1
at ./pkg/owner/manager.go:300
21 0x00000000020c2721 in runtime.goexit
at /home/genius/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700
```

Similiar to https://github.com/pingcap/tidb/issues/58418

### 4. What is your TiDB version? (Required)

master
70393b5e7ec375dbc25d6bde3bce3bbf01cb877d

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.