ClickHouse / ClickHouse/ClickBench

concurrent_qps measures swap-thrashing, not throughput, for single-process engines (introduced in #860)

Open
#946 3 comments 1 reaction 1 assignee Claimed by @george-larionov View on GitHub
Dominant language
Shell
Stars
1.1k
Forks
313
Avg merge
2h 43m
Merged PRs (30d)
598

Description

## Summary

The `concurrent_qps` test introduced in #860 produces misleading, non-reproducible numbers for single-process / in-process engines (embedded CLIs and dataframe-server systems). For these systems it spawns N independent full-machine processes that collectively oversubscribe RAM, so on memory-constrained instances the metric measures **swap-thrashing throughput**, not query throughput — and whether it completes at all depends on the runner environment rather than the engine.

This surfaced in #943 (DataFusion), but the root cause is in `lib/benchmark-common.sh` (`bench_concurrent_qps`) and affects a whole class of systems.

## Evidence

`bench_concurrent_qps` launches `BENCH_CONCURRENT_CONNECTIONS` (default **10**) workers for `BENCH_CONCURRENT_DURATION` (default **600s**). For `BENCH_DURABLE=no` engines like `datafusion-cli`, each worker's `./query` spawns a fresh process that assumes it owns the whole machine. 10 × ~7 GB RSS ≈ 70 GB against 32 GB RAM.

Checked-in DataFusion (single) results, `concurrent_qps` vs. machine RAM:

| Machine | vCPU / RAM | concurrent_qps | err |
|---|---|---|---|
| c6a.xlarge | 4 / 8 GB | 0.025 | 0 |
| c6a.2xlarge | 8 / 16 GB | 0.042 | 0 |
| c6a.4xlarge | 16 / 32 GB | 0.11 | 0 |
| c8g.4xlarge | 16 / 32 GB | 0.092 | 0 |
| c6a.metal | 192 / 384 GB | 4.017 | 0 |
| c8g.metal-48xl | 192 / 384 GB | 5.388 | 0 |
| c7a.metal-48xl | 192 / 384 GB | 5.888 | 0 |

(`datafusion-partitioned` shows the same shape: 0.025–0.097 on the small boxes, 4.9–7.0 on the metals.)

The QPS tracks **RAM**, not core count — the metric on the sub-RAM machines is reporting how fast the box can page, not how fast the engine runs queries. `err=0` everywhere, so nothing crashed in the automated runs; the processes just crawled through swap.

## Why the same instance type gives different outcomes

`cloud-init.sh.in` adds a **16 GB swapfile** and installs **earlyoom** on launch. In the automated fleet run a c6a.4xlarge therefore *thrashes through swap and finishes* at ~0.11 qps. A contributor running `./benchmark.sh` on a **bare** c6a.4xlarge (no swap) gets the kernel OOM-killing `datafusion-cli` outright (see #943). So a *published* number depends on the presence of swap in the runner environment — a reproducibility problem independent of any one engine.

## Scope

The same dynamic applies to every `BENCH_DURABLE=no` single-process / in-process system where N independent processes oversubscribe RAM — e.g. duckdb (and dataframe/memory variants), sqlite, hyper, chdb, pandas, polars-dataframe, daft, sirius. Any of these on a sub-RAM machine will show the same artifact.

## Options

1. **Run the concurrent test at concurrency = 1 for single-process engines.** Measures real serial throughput, stays comparable, never oversubscribes RAM. (Suggested by @alamb in #943.)
2. **Disable `concurrent_qps` for the whole single-process / in-process class.** #943 does this for DataFusion only (`BENCH_CONCURRENT_DURATION=0`); make it class-wide so the dashboard isn't uneven (one embedded engine reporting `null` while peers report swap artifacts).
3. **Add a "fit in RAM" guard** analogous to the partial-load `>= 5 GB` check: if the concurrent test's working set exceeds available RAM, record `null` instead of a thrashing number.
4. **Pin the environment**: document/encode whether swap is part of the canonical setup so the same instance type reproduces the same result. (Weakest on its own — a number that only completes because of swap shouldn't be published regardless.)

Recommendation: option 1 or 2 for the single-process class (subsumes #943 into a consistent, class-wide fix), plus nulling the existing swap-artifact `concurrent_qps` values already checked in for these systems on the RAM-constrained machines.

## Related

- #860 — introduced the concurrent QPS test
- #943 — DataFusion-specific workaround that prompted this

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.