matrixorigin / matrixorigin/matrixone

[Bug]: single node CN shutdown now take quite long

Open
#20,731 0 comments 0 reactions 1 assignee Claimed by @daviszhen View on GitHub
ai-medium kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [X] I have checked the existing issues.

### Branch Name

2.0-dev, main

### Commit ID

42141a8fe6ced1853d75e2939545de1f9a5b11fa

### Other Environment Information

```Markdown
- Hardware parameters:
- OS type:
- Others:
```

### Actual Behavior

Hit ^C to stop a one node deployment. It takes quite sometime to shutdown. Some sample log. At the end the system WILL correctly shutdown, but given that a single node instance has really nothing to migrate, it should be faster.

```
{"level":"ERROR","time":"2024/12/11 17:15:36.252835 -0800","name":"cn-service.lock-client","caller":"morpc/backend.go:251","msg":"connect to remote failed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","remote":"127.0.0.1:19002","backend-id":"0193b86f-83bb-795a-bcd8-047caf5c8748"}
{"level":"ERROR","time":"2024/12/11 17:15:36.252853 -0800","name":"cn-service.lock-client.lock-client","caller":"morpc/client.go:515","msg":"create backend failed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","backend":"127.0.0.1:19002","error":"can not connect to remote backend, dial tcp4 127.0.0.1:19002: connect: connection refused"}
{"level":"ERROR","time":"2024/12/11 17:15:36.252880 -0800","name":"cn-service.lockservice","caller":"lockservice/lock_table_keeper.go:197","msg":"failed to keep lock table bind","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","error":"can not connect to remote backend, dial tcp4 127.0.0.1:19002: connect: connection refused"}
{"level":"ERROR","time":"2024/12/11 17:15:37.254325 -0800","name":"cn-service.lock-client","caller":"morpc/backend.go:882","msg":"init remote connection failed, retry later","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","remote":"127.0.0.1:19002","backend-id":"0193b86f-87a5-7785-8ec0-2baabf4742f8","can-retry":false,"error":"dial tcp4 127.0.0.1:19002: connect: connection refused"}
...

{"level":"INFO","time":"2024/12/11 17:15:43.468137 -0800","name":"cn-service.lockservice","caller":"lockservice/deadlock.go:130","msg":"dead lock checker","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","cost":"28m25.433718667s"}
{"level":"INFO","time":"2024/12/11 17:15:43.473217 -0800","name":"cn-service.lockservice","caller":"lockservice/lock_table_keeper.go:83","msg":"keep lock table bind task","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","cost":"28m25.437829875s"}
{"level":"INFO","time":"2024/12/11 17:15:43.473246 -0800","name":"cn-service.lockservice","caller":"lockservice/lock_table_keeper.go:103","msg":"keep remote locks task","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","cost":"28m25.437857875s"}
{"level":"INFO","time":"2024/12/11 17:15:43.514240 -0800","name":"cn-service.gossip","caller":"gossip/node.go:195","msg":"leaving gossip cluster","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","node ID":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf"}
{"level":"INFO","time":"2024/12/11 17:15:43.524881 -0800","name":"cn-service","caller":"logutil/action.go:38","msg":"close components completed","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","action":"close","components":"cnservice","cost":"1m52.866069625s"}
{"level":"INFO","time":"2024/12/11 17:15:43.525003 -0800","caller":"mo-service/main.go:471","msg":"Shutdown service complete."}
{"level":"INFO","time":"2024/12/11 17:15:43.799432 -0800","caller":"motrace/trace.go:225","msg":"Shutdown trace complete."}
{"level":"INFO","time":"2024/12/11 17:15:43.803384 -0800","caller":"mo-service/main.go:126","msg":"Shutdown complete"}
```

### Expected Behavior

_No response_

### Steps to Reproduce

```Markdown
Start single node instance, hit ^C to shutdown.
```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.