matrixorigin / matrixorigin/matrixone

[Performance]: Standalone BVT ~1.77× slower on arm64-mo-shanghai-8c16g vs ubuntu-22.04 (median script ~2.5×)

Open
#25,116 3 comments 0 reactions 1 assignee Claimed by @VioletQwQ-0 View on GitHub
ai-heavy deferred kind/performance
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

CI A/B validation ([ci-test run 28020049513](https://github.com/matrixorigin/ci-test/actions/runs/28020049513)) compared **Standalone BVT Pessimistic** on the **same** `matrixorigin/matrixone@main` commit, same mo-tester flags (`-e optimistic -s resources`), only runner/arch differs:

| | ARM | x86 |
|---|-----|-----|
| Runner | `arm64-mo-shanghai-8c16g` (8 CPU / 16 GiB K8s pod) | `ubuntu-22.04` GitHub hosted |
| Job | [82933895760](https://github.com/matrixorigin/ci-test/actions/runs/28020049513/job/82933895760) | [82933819100](https://github.com/matrixorigin/ci-test/actions/runs/28020049513/job/82933819100) |
| **Start BVT Test wall time** | **59.5 min** | **33.7 min** |
| **Ratio** | | **1.77×** |

Both jobs **passed** (functional correctness OK). **Build** on the same ARM runner was **faster** than x86 in the same run (~5.8 min vs ~7.0 min), so the gap is **MO runtime / SQL execution**, not compile time.

Artifacts: `bvt-report-arm-pessimistic` vs `bvt-report-x86-pessimistic` (1009 scripts each).

---

## Environment

- **MatrixOne**: `matrixorigin/matrixone` branch `main` (ci-test workflow inputs)
- **mo-tester**: `main`, `./run.sh -n -g -o -p .../cases -s .../resources -e optimistic`
- **ARM runner**: `arm64-mo-shanghai-8c16g`, Shanghai self-hosted K8s pod
- **x86 runner**: `ubuntu-22.04` GitHub Actions hosted
- **Validation workflow**: [ARM Runner Migration Test](https://github.com/matrixorigin/ci-test/actions/runs/28020049513) with `run_x86_baselines=true`

---

## mo-tester aggregate (from `report.txt`)

| Metric | ARM | x86 | Ratio |
|--------|-----|-----|-------|
| Total script time (1009 scripts) | 3549.8 s (~59.2 min) | 2010.1 s (~33.5 min) | **1.77×** |
| Report `[SUMMARY] COST` | 3175 s | 1710 s | 1.86× |
| **Median per-script ratio** | | | **2.53×** |
| Mean per-script ratio | | | 2.61× |

### Per-script ratio distribution (n=1003)

| Bucket | Count |
|--------|-------|
| ARM faster (<1.0×) | 35 |
| 1.0–1.5× | 124 |
| 1.5–2.0× | 179 |
| 2.0–3.0× | 277 |
| ≥3.0× | 388 |

**Not a few outliers**: ~746 scripts with x86 duration **<1 s** still have **median 2.63×** on ARM. Slowdown is broad-based on short/medium SQL cases.

### Slowdown by x86 script duration

| x86 bucket | Scripts | Aggregate ARM/x86 | Median ratio |
|------------|---------|-------------------|--------------|
| <1 s | 746 | 2.67× | 2.63× |
| 1–5 s | 170 | 2.39× | 2.38× |
| 5–30 s | 78 | 1.89× | 2.09× |
| ≥30 s | 9 | **0.94×** | **1.02×** |

Long **fixed-wait** cases (sleep/cron/restore polling) are ~1.0×; they dilute job-level average but do not explain the gap.

Examples (ratio ≈ 1.0×): `function/func_sleep.sql`, `pessimistic_transaction/vector/vector_ivf_restore_idxcron.sql` (~105 s both sides).

---

## Top categories by ARM extra time

| Category | ARM total | x86 total | Aggregate ratio |
|----------|-----------|-----------|-----------------|
| `snapshot/` | 936 s | 321 s | **2.92×** |
| `tenant/` | 96 s | 32 s | **3.02×** |
| `join/` | 89 s | 30 s | **2.96×** |
| `zz_accesscontrol/` | 123 s | 45 s | **2.75×** |
| `optimizer/` | 170 s | 69 s | **2.46×** |
| `pessimistic_transaction/` | 850 s | 718 s | 1.18× (many fixed-wait vector/fulltext cases) |

## Top single-script contributors to ARM − x86 delta (~25.7 min total)

| Extra time | Ratio | Script |
|------------|-------|--------|
| +65 s | 3.59× | `optimizer/shuffle.test` |
| +47 s | 2.99× | `join/spill.sql` |
| +29 s | 2.51× | `pessimistic_transaction/vector/vector_hnsw_f64.sql` |
| +25 s | 3.10× | `snapshot/restore_dropped_account.sql` |
| +24 s | 2.52× | `zz_accesscontrol/create_account.sql` |

ARM faster example: `fulltext/gojieba.sql` 60 s vs 102.8 s (0.58×).

---

## Impact

- **CI**: Standalone / Upgrade BVT jobs are **not suitable** for `arm64-mo-shanghai-8c16g` today (~2× wall time); plan keeps BVT on `ubuntu-22.04`.
- **Product**: If ARM64 is a delivery target, ~2.5× median SQL case slowdown needs investigation before perf parity with amd64.

## Initial assessment

- **Not observed as functional bug** (BVT passed; fixed-wait cases match).
- Likely **arm64 runtime / query-path performance gap** (optimization debt), not mo-tester or CI framework overhead.

## Suggested next steps

1. Profile on ARM vs x86 for top delta scripts: `optimizer/shuffle.test`, `join/spill.sql`, `snapshot/restore_dropped_account.sql`.
2. Check ARM runner CPU features (CRC32/AES etc.) affecting hash hot paths (`pkg/container/hashtable/hash_arm64.go`).
3. Track arm64 BVT duration as a perf metric if ARM CI coverage expands later.

---

## References

- ci-test run: https://github.com/matrixorigin/ci-test/actions/runs/28020049513
- ARM BVT job: https://github.com/matrixorigin/ci-test/actions/runs/28020049513/job/82933895760
- x86 BVT job: https://github.com/matrixorigin/ci-test/actions/runs/28020049513/job/82933819100

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.