matrixorigin / matrixorigin/matrixone

[Bug]: ClickBench Q30 is significantly slower in MatrixOne than DuckDB

Open
#27,682 1 comment 0 reactions 1 assignee Claimed by @XuPeng-SH View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Parent issue

This is a sub-issue of #27665: [significant gap in ClickBench query performance between MatrixOne and DuckDB](https://github.com/matrixorigin/matrixone/issues/27665).

## Summary

ClickBench **Q30** shows a significant MatrixOne performance gap against the DuckDB reference. The values below are elapsed seconds. Each result array is the three measurements emitted by the adapter: `[run 1, run 2, run 3]`; `warm median` is the median of the available run-2/run-3 values.

## Query

```sql
SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits;
```

## Performance comparison

| Engine | Run 1 | Run 2 | Run 3 | Warm median |
| --- | ---: | ---: | ---: | ---: |
| MatrixOne | 10.130 | 8.080 | 8.110 | 8.095 |
| DuckDB | 0.174 | 0.067 | 0.070 | 0.069 |

- MatrixOne / DuckDB warm-median ratio: **118.18x**
- Warm-median difference: **8.026 seconds**
- Raw MatrixOne timing array: `[10.130, 8.080, 8.110]`
- Raw DuckDB timing array: `[0.174, 0.067, 0.070]`

## Environment and data

MatrixOne measurement:

- Branch: `main`
- Commit: `249a458ba567b26d1818fb0496b449135cb05ef2`
- Hardware: `server 129`
- OS: CentOS
- Dataset: ClickBench `hits`, **99,997,497 rows**
- MatrixOne raw output: `/mnt/fastdata/ClickBench/matrixone/raw_results.md`

DuckDB reference:

- Result file: `duckdb/results/20260511/c6a.metal.json`
- Machine label: `c6a.metal`
- Result date: `2026-05-11`

The two measurements were not collected on the same hardware. The DuckDB value is the latest local `c6a.metal` reference available; an apples-to-apples rerun on the same server is recommended before assigning a final performance regression magnitude.

## Steps to reproduce

1. Follow the reproduction steps in [#27665](https://github.com/matrixorigin/matrixone/issues/27665), including the MatrixOne and DuckDB adapter READMEs:
- [MatrixOne ClickBench README](https://github.com/matrixorigin/mo-benchmark/blob/main/clickbench/matrixone/README.md#run-in-129)
- [DuckDB ClickBench README](https://github.com/matrixorigin/mo-benchmark/blob/main/clickbench/duckdb/README.md#%E8%BF%90%E8%A1%8C%E5%AE%8C%E6%95%B4-clickbench)
2. Use the same `hits.tsv.gz` dataset for both engines (the adapters support loading the gzip file directly).
3. In the MatrixOne adapter, run `./benchmark.sh` after setting `MATRIXONE_HOME`, `MATRIXONE_DATA_FILE`, `MATRIXONE_HOST`, `MATRIXONE_PORT`, `MATRIXONE_USER`, and `MATRIXONE_PASSWORD` as described in #27665.
4. In the DuckDB adapter, run `./benchmark.sh` after setting `DUCKDB_BIN_PATH`, `DUCKDB_DB_PATH`, and `DUCKDB_DATA_PATH` as described in the README.
5. Compare the Q30 entry in `queries.sql`; run the query three times after loading the data and record elapsed time and whether the query completes successfully.

## Notes

- The timing gap is the subject of this sub-issue; query-result correctness should be verified independently.

## Expected behavior

On comparable hardware with the same dataset, MatrixOne should not exhibit a large query-level performance gap for this workload. Please investigate the execution plan, operators, memory use, and any spill or connection/timeout behavior relevant to this query.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.