matrixorigin / matrixorigin/matrixone
fix: quiesce and drain TN RPC handlers before storage close
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same bug?
- [x] I have checked the existing issues. Parent issue #27945 tracks the standalone clean-restart failure; this issue tracks the distinct TN RPC drain invariant required by that lifecycle.
### Branch Name
main
### Commit ID
`c46d897e9645b80178568ef0783dd8e99e527222`
### Other Environment Information
- Hardware parameters: mo-55 bare-metal validation host; resource thresholds remained above 12 GiB MemAvailable and 100 GiB root free space.
- OS type: Linux amd64.
- Deployment: single-process standalone containing CN, TN, LogService, and HAKeeper.
- Workload: retained 2,303,643-row FULLTEXT2 runtime with two accepted TN transactions in flight during `SIGTERM`.
- Parent lifecycle issue: #27945.
- Implementation PR: #28016, logically dependent on PR #28015.
### Actual Behavior
During standalone graceful shutdown, accepted TN transaction RPC handlers can remain active while TN replicas or storage are being cancelled and while the process-level shutdown advances toward LogService shutdown.
In the captured failure, exactly two TN RPC workers remained inside the transaction commit path:
```text
TN transaction RPC worker
→ TAE Txn.commit1PC()
→ txnStore.WaitWalAndTail()
→ WAL entry.WaitDone()
→ LogService append
```
The TN RPC server does not expose a lifecycle boundary that first rejects new producers and then drains already accepted handlers. Queued, local-dispatch, forwarded, and active requests therefore do not share one sealed accounting model, and shutdown cannot prove that every accepted handler reached a terminal state before replica/storage dependencies are closed.
This can make a dependency-ordered standalone shutdown stall or destroy resources still needed by accepted commits. It also makes repeated or concurrent close behavior difficult to reason about.
### Expected Behavior
TN transaction RPC shutdown should follow a private, bounded lifecycle:
```text
Quiesce
→ Drain accepted handlers
→ Close replicas and storage
```
After the quiesce linearization point, network and local-dispatch producers must be rejected and the accepted-handler count must not increase. Queued requests must be cancelled and released exactly once. Active handlers must retain replica, storage, and WAL dependencies until they reach a terminal state.
If drain reaches its deadline, shutdown must return a deterministic error without destroying dependencies that active handlers still require. A commit that has entered the durability stage must not be reported as rolled back merely because the client disconnected or shutdown began.
### Steps to Reproduce
1. Start a standalone MatrixOne process containing CN, TN, LogService, and HAKeeper.
2. Establish at least two transaction RPC requests and pause or delay them after the server has accepted them but before their WAL durability terminal is published.
3. Send `SIGTERM` to begin graceful standalone shutdown.
4. Observe TN shutdown while the accepted RPC handlers are still active.
5. Without an explicit quiesce/drain boundary, new/local producers can race with shutdown and replica/storage cancellation can overlap the accepted handlers.
6. Confirm the terminal wait chain with the TN RPC worker and WAL waiter goroutine stacks.
The deterministic regression coverage for the fix should exercise queued, forwarded-wait, local-dispatch, handler-active, and durability-stage requests, including repeated close and drain timeout.
### Additional information
- Scope: private TN and transaction-RPC lifecycle only.
- Non-goals: no SQL, DDL, public RPC interface, transaction wire protocol, persisted format, owner timeout, discovery timeout, or transaction-outcome semantic change.
- PR dependency: PR #28015 must establish process-level dependency ordering first. PR #28016 must be rebased to a B-only diff after PR #28015 merges.
- BVT: N/A. SQL BVT cannot deterministically stop an RPC at queued/active/durability boundaries while delivering process shutdown and asserting internal dependency lifetime. Package and race tests must cover those terminals, followed by an A+B in-flight-commit standalone test.
- QA required: yes. The production terminal is an accepted transaction during graceful shutdown; exact-head in-flight-commit E2E remains required before closure.
Contributor guide
Assessment
This issue has not been assessed yet.