matrixorigin / matrixorigin/matrixone

[Bug]: Sustained 1 KiB-row REPLACE falls from 400 to 263 rows/s and hits 111–116s failures on Standard v4.1.0

Open
#27,403 5 comments 0 reactions 1 assignee Claimed by @ck89119 View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

在 JED 的 managed Standard v4.1.0 实例上执行 72 小时大规模持续写入时,批量 `REPLACE` 无法维持配置的 `400 rows/s`。随着运行时间和表规模增长,累计吞吐下降到约 `263 rows/s`,最近约 4.5 小时只有约 `134 rows/s`,并出现两条持续 111–116 秒后以 `broken pipe` 失败的写语句。

相同 workflow、相同 workload 参数在 MTY v4.1.1 新实例上累计约 `398 rows/s`,可达到目标。该对照说明 `400 rows/s` 是客户端实际施加的目标,而不是从 JED 当前吞吐反推得到的配置值。

本 Issue 记录 managed v4.1.0 现场的大规模持续写入退化,供定位 TN/merge/checkpoint/commit 或 `REPLACE` 路径的瓶颈;尚未声称已在 official `main` 复现。

## Environment

- Affected server: `8.0.30-OmniFabric-v4.1.0`
- Deployment: managed Standard, JED
- Topology visible to tenant: two CN endpoints behind the managed database entry
- Observation window: `2026-08-18 10:40:13` to `2026-08-20 09:32:47` UTC
- Workload source: `matrixorigin/cloudsigma-gitops` commit `25933d06e72567a559422ed0eca05315c9df5da9`
- Latest official `main` reference at filing time: `94528dcb63d55921b936cda611ea61eb1862ea79`
- `main` status: not reproduced yet; the SHA above is a version boundary, not a tested build

Endpoints, tenant/account identifiers, credentials, node addresses and payload values are intentionally omitted.

## Workload

The attached 72-hour stability job uses:

```text
rows_per_second: 400
rows_per_batch: 1000
payload_extra_bytes_per_row: 1024
business_query_interval_seconds: 0
statement_observer_interval: 900s
writer connections: 1
write SQL: REPLACE INTO
primary key: (run_id, sequence_id)
```

Each run uses a new `run_id`, and `sequence_id` increases from zero. Retried batches are idempotent because the composite primary key is deterministic.

## Steps to reproduce

1. Attach the workload above to an existing Standard instance whose target table already contains tens of millions of rows from earlier soak runs.
2. Start a new 72-hour run at `400 rows/s`, `1000 rows/batch`, with 1024 additional payload bytes per row.
3. Periodically read the maximum `sequence_id` for the current `run_id` and compare it with `elapsed_seconds * 400`.
4. Aggregate the tagged `REPLACE` records in `system.statement_info` by time window, recording count, average/max duration, status and error.
5. Continue beyond 40 hours and observe whether the client can keep up with the target.

## Actual behavior

At `2026-08-20 09:31:37` UTC:

```text
elapsed: ~46.86 hours
latest sequence_id: 44,347,999 (~44.348 million rows)
rows expected at 400 rps: ~67.47 million
cumulative rate: ~263 rows/s
target attainment: ~65.7%
```

The degradation continued during the observation:

```text
earlier snapshot: 42.158 million rows at 2026-08-20 04:58:39 UTC
later snapshot: 44.348 million rows at 2026-08-20 09:31:37 UTC
interval rate: ~134 rows/s over ~4.55 hours
```

The server-side `REPLACE` latency was not monotonically increasing, but its baseline and tail became materially worse:

```text
initial windows average: ~0.085–0.122s per recorded REPLACE statement
later 6h windows average: ~0.299–0.415s
2026-08-20 04:00–09:32 UTC:
statements: 6,010
average duration: ~0.971s
max duration: 115.638s
failed statements: 2
```

The two failures were:

```text
duration=115.638s, err_code=20101, routine send response failed: broken pipe
duration=111.599s, err_code=20101, routine send response failed: broken pipe
```

Later writes succeeded and the Action remained running, so this was recovery after long stalls rather than a permanent stop. However, retry/recovery did not restore the configured cumulative throughput.

## Expected behavior

- A supported Standard instance should sustain at least the workload's 95% acceptance threshold (`>=380 rows/s`) for this bounded soak, or expose an explicit capacity/admission error instead of silently falling behind.
- Batch latency should remain bounded without 110+ second server executions and socket failures.
- Temporary retryable failures should recover without creating an ever-growing schedule backlog.
- Data must remain contiguous and idempotent after retry/reconnect.

## Stability and controls

- Affected JED run: one long-running field observation; the issue is not claiming 3/3 reproduction on official `main`.
- MTY control: the same workflow commit and workload parameters on a fresh v4.1.1 Standard instance reached approximately `29.324 million` rows in `20.45 hours`, or about `398 rows/s` (`~99.6%` of target). A 12-minute metadata delta was approximately `409 rows/s`.
- The MTY control differs in release (`v4.1.1`) and pre-existing table history, so it proves that the workload generator can reach the target but does not isolate the product change or physical-table-state trigger.
- The deterministic key makes retries idempotent. Full sequence-gap/checksum validation will run at finalization; it was not executed while the soak was active.
- No DML/DDL was issued during investigation other than the workload itself; diagnosis used read-only metadata and `statement_info` queries.

## Evidence

- Affected JED soak: https://github.com/matrixorigin/cloudsigma-gitops/actions/runs/32127762306
- MTY control: https://github.com/matrixorigin/cloudsigma-gitops/actions/runs/32231917947

Both Actions were still in progress at the latest snapshot. The numbers above come from the live target tables and redacted `system.statement_info` aggregates, not from the Action status alone.

## Code analysis

Confirmed at the workload boundary:

- The scheduler targets one 1,000-row batch every 2.5 seconds.
- When it falls behind, it advances `next_write` by the fixed interval and immediately tries subsequent batches; therefore the cumulative shortfall is not caused by a deliberate reduction of the configured target.
- The writer is single-threaded, so prolonged batch execution/reconnect/retry directly limits catch-up capacity.
- The server retained two failed `REPLACE` executions for 111–116 seconds and then failed while sending the response to a client whose socket was already gone.

Root-cause hypothesis, not yet proven: the large existing table/object history plus shared TN background pressure (merge/checkpoint/commit/logtail) increases `REPLACE` service time until the single writer cannot sustain or catch up with 400 rows/s. Data-plane metrics and logs are required to distinguish that from a `REPLACE` conflict-detection/flush path bottleneck.

## Regression coverage

Add a big-data/stability regression rather than a BVT timing assertion:

1. Preload the same composite-PK table to at least 50 million rows or an equivalent object/L0 history.
2. Run 1 KiB payload writes at 400 rows/s for a long enough window to cross merge/checkpoint cycles.
3. Assert achieved throughput `>=95%` of target in every bounded window and cumulatively.
4. Record batch p50/p95/p99/max, retry/reconnect counts, zero-write windows and schedule lag.
5. Validate sequence continuity, checksum/idempotency and post-reconnect correctness.
6. Correlate CN/TN/LogService memory, restart/OOM, merge, checkpoint, commit, logtail and object-store metrics.

## Related

- #27046 tracks single-row watermark `REPLACE` waiting in `Multi Update` under shared background pressure. This issue tracks end-to-end sustained batch-write capacity and target attainment; the two may share a TN/merge trigger.
- #27045 tracked table-scale read amplification for new-key `INSERT` when merge was paused; it did not cover this batch `REPLACE` throughput acceptance failure.
- #27059 tracked slow `DROP/TRUNCATE` under the same class of shared background pressure; it did not cover long-running DML throughput.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.