matrixorigin / matrixorigin/matrixone

Enhancement: Implement Transaction-Level Lock Count Accumulation for Lock Escalation

Open
#23,494 0 comments 0 reactions 1 assignee Claimed by @iamlinjunhong View on GitHub
kind/enhancement priority/p0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Problem Statement

Currently, MatrixOne's lock escalation mechanism only evaluates each DML operation independently at compile time based on statistics. This creates several issues:

1. **Multi-Statement Transaction Memory Explosion**: When a transaction executes multiple INSERT/UPDATE/DELETE operations, each is evaluated independently without accumulating lock counts. A transaction with 100,000 individual INSERT statements (each inserting 1 row) will never escalate to table lock, even though it accumulates 100,000 row locks.

2. **Inaccurate Statistics**: Lock escalation decisions rely entirely on compile-time statistics. When statistics are outdated or inaccurate, the system may fail to escalate locks even when actual affected rows far exceed the threshold.

3. **No Runtime Escalation**: The execution phase only counts `lockCount` but never triggers escalation. This means the system cannot adapt to runtime conditions.

## Objectives

Implement a comprehensive lock escalation enhancement strategy:

1. **Transaction-Level Lock Count Accumulation**: Track total lock count across all operations within a transaction and escalate to table lock when threshold is exceeded.

2. **Enhanced Statistics Estimation**: Improve NDV (Number of Distinct Values) estimation for more expression types:
- Division operations: `col / N`
- Date functions: `YEAR(date_col)`, `MONTH(date_col)`
- IN clauses: `col IN (...)`
- Range predicates: `col BETWEEN a AND b`

3. **Monitoring and Alerting**: Add metrics to track per-transaction lock counts and alert when approaching thresholds.

## Expected Benefits

- Prevent memory explosion in multi-statement transactions
- Improve robustness against inaccurate statistics
- Better resource utilization through automatic lock escalation
- Reduced lock management overhead for large operations

## Related Issues

- https://github.com/matrixorigin/matrixone/issues/23483
- https://github.com/matrixorigin/matrixone/issues/21608

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.