matrixorigin / matrixorigin/matrixone
[Bug] AI-WALK (Txn workspace): repeated mutations degrade super-linearly in long transactions
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Is there an existing issue for the same bug?
- [x] I searched open and closed issues for transaction workspace scaling, repeated updates, and the `AI-WALK` title prefix. I did not find the same report.
## Branch Name
`main`
## Commit ID
`efe2379669c61a0a7b1a34dd9bf94e59bcf61132` (2026-07-10 21:00:24 +0800)
## Other Environment Information
- Hardware: 20 vCPU (Intel Core i9-10900K), 62 GiB RAM
- OS: Debian GNU/Linux 12 (bookworm), Linux 6.1.0-31-amd64, x86_64
- Docker server: 29.1.3
- MO image: source-built `matrixorigin/matrixone:workspace-attack-efe2379669`, image ID `sha256:5de7da05e3adb305b7cf1a35b8619c6e032c10f083af3a1fd2a1de0224b1c9e8`
- Deployment: isolated local-disk Docker cluster with 1 Log Service, 1 TN, 2 CNs and 1 proxy; test traffic used CN1 on port 26001
- Profiling: CN started with `-block-profile-rate=100 -mutex-profile-fraction=100 -debug-http=:6061`; CPU profile captured from port 26161
## Actual Behavior
Repeatedly updating the same primary-key row as separate statements in one explicit transaction becomes progressively slower as transaction history grows, even though the logical working set is one row.
| UPDATE statements | Mean/statement | First-quarter mean | Last-quarter mean | Commit time |
|---:|---:|---:|---:|---:|
| 5,000 | 0.647 ms | 0.474 ms | 0.810 ms | 5.5 ms |
| 10,000 | 0.881 ms | 0.535 ms | 1.236 ms | 10.2 ms |
| 20,000 | 1.403 ms | 0.648 ms | 2.236 ms | 20.3 ms |
| 30,000 | 2.177 ms | 0.770 ms | 3.991 ms | 41.6 ms |
At 30,000 statements, the last quarter is 5.18x slower than the first quarter. Commit itself takes only 41.6 ms, so the dominant regression occurs during statement execution.
The same history-dependent pattern appears in other workspace mutations:
- 10,000 REPLACE statements: last/first-quarter latency ratio 2.56x.
- 5,000 DELETE+INSERT pairs: last/first-quarter latency ratio 1.83x.
Functional results were correct in the tested range; this issue is transaction scalability and CPU amplification.
## Expected Behavior
When thousands of statements repeatedly supersede the same row/primary key, per-statement cost should remain approximately stable or grow sublinearly. It should be bounded by the current table/row working set, not by the entire superseded statement history.
## Steps to Reproduce
```python
import statistics
import time
import pymysql
N = 30000
c = pymysql.connect(host="127.0.0.1", port=26001,
user="root", password="111", autocommit=False)
cur = c.cursor()
cur.execute("drop database if exists ai_walk_workspace_perf")
cur.execute("create database ai_walk_workspace_perf")
cur.execute("use ai_walk_workspace_perf")
cur.execute("create table point_t(id bigint primary key, v bigint)")
cur.execute("insert into point_t values(1, 0)")
c.commit()
cur.execute("begin")
latency_ms = []
for _ in range(N):
started = time.perf_counter()
cur.execute("update point_t set v=v+1 where id=1")
latency_ms.append((time.perf_counter() - started) * 1000)
c.commit()
q = N // 4
print("mean", statistics.mean(latency_ms))
print("first-quarter", statistics.mean(latency_ms[:q]))
print("last-quarter", statistics.mean(latency_ms[-q:]))
cur.execute("select v from point_t where id=1")
assert cur.fetchone()[0] == N
```
Repeat with 5k, 10k, 20k, and 30k statements. For a server-side profile, capture CN CPU while the 30k loop runs:
```bash
curl -o workspace-update.pprof 'http://127.0.0.1:26161/debug/pprof/profile?seconds=30'
go tool pprof -top ./mo-service workspace-update.pprof
```
## Additional information
The 30-second CN CPU profile attributed substantial time to transaction-wide/history scans:
```text
Transaction.IncrStatementID 6.67s cumulative
Transaction.mergeTxnWorkspaceLocked 3.23s flat / 3.81s cumulative
LocalDisttaeDataSource.filterInMemUnCommittedInserts 1.80s flat / 6.50s cumulative
Transaction.forEachTableHasDeletesLocked 2.73s flat
Transaction.ForEachTableWrites 1.35s flat / 3.46s cumulative
LocalDisttaeDataSource.workspaceDeleteEntriesLocked 2.22s flat / 2.56s cumulative
```
Source inspection shows several statement/read paths scanning `txn.writes` or rebuilding data from `writes[:txnOffset]`. Each UPDATE grows transaction history, and later statements repeatedly revisit that history, producing near-quadratic aggregate work.
Suggested solution direction: maintain table-scoped incremental workspace indexes and compact superseded versions of the same row/PK at write or statement boundaries. Datasource reads, delete enumeration, and tombstone transfer should consume shared indexes instead of independently rescanning the transaction-wide write slice.
Contributor guide
Assessment
This issue has not been assessed yet.