matrixorigin / matrixorigin/matrixone
Enhancement: Implement Transaction-Level Lock Count Accumulation for Lock Escalation
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Problem Statement
Currently, MatrixOne's lock escalation mechanism only evaluates each DML operation independently at compile time based on statistics. This creates several issues:
1. **Multi-Statement Transaction Memory Explosion**: When a transaction executes multiple INSERT/UPDATE/DELETE operations, each is evaluated independently without accumulating lock counts. A transaction with 100,000 individual INSERT statements (each inserting 1 row) will never escalate to table lock, even though it accumulates 100,000 row locks.
2. **Inaccurate Statistics**: Lock escalation decisions rely entirely on compile-time statistics. When statistics are outdated or inaccurate, the system may fail to escalate locks even when actual affected rows far exceed the threshold.
3. **No Runtime Escalation**: The execution phase only counts `lockCount` but never triggers escalation. This means the system cannot adapt to runtime conditions.
## Objectives
Implement a comprehensive lock escalation enhancement strategy:
1. **Transaction-Level Lock Count Accumulation**: Track total lock count across all operations within a transaction and escalate to table lock when threshold is exceeded.
2. **Enhanced Statistics Estimation**: Improve NDV (Number of Distinct Values) estimation for more expression types:
- Division operations: `col / N`
- Date functions: `YEAR(date_col)`, `MONTH(date_col)`
- IN clauses: `col IN (...)`
- Range predicates: `col BETWEEN a AND b`
3. **Monitoring and Alerting**: Add metrics to track per-transaction lock counts and alert when approaching thresholds.
## Expected Benefits
- Prevent memory explosion in multi-statement transactions
- Improve robustness against inaccurate statistics
- Better resource utilization through automatic lock escalation
- Reduced lock management overhead for large operations
## Related Issues
- https://github.com/matrixorigin/matrixone/issues/23483
- https://github.com/matrixorigin/matrixone/issues/21608
Contributor guide
Assessment
This issue has not been assessed yet.