google / google/capsem

[triage:tooling-04-no-warmup-and-spawn-skew-in-throughput] Throughput timing includes worker-spawn skew and has no warmup; first-request connection setup is counted

Open
#148 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

Imported from Capsem triage report `tooling-04-no-warmup-and-spawn-skew-in-throughput.md`.

- Severity: `low`
- Category: `bug (measurement correctness)`
- Area: `capsem-bench`
- Location: `crates/capsem-bench/src/main.rs:583-607` (`run_http_scenario`), `617-668` (`run_dns_scenario`)
- Confidence: `verified`

## Summary
Each scenario's wall clock starts *before* the workers are spawned and stops after all join. There is no warmup phase, so TCP connect + TLS handshake for the first request on each pooled connection, plus tokio task-spawn scheduling latency, is folded into both per-request latency samples and the scenario-level `requests_per_sec` / `bytes_per_sec`. For low request counts or high-latency lanes (the guest-through-Capsem lane), cold-connection cost measurably distorts the host-vs-guest delta the tool exists to compute.

## Evidence
`run_http_scenario` (lines 583-599):
```rust
let started = Instant::now();
let tasks = (0..workers).map(|idx| { ... tokio::spawn(async move { ... }) });
let joined = try_join_all(tasks).await...;
let wall_time = started.elapsed();
```
`started` is captured before the `tokio::spawn` closures are constructed and scheduled; `wall_time` therefore includes spawn/scheduling skew across `workers` tasks. The first `run_one_request` per worker (line 591) performs `request.send()` (line 754) on a fresh pooled connection — connect + TLS for HTTPS lanes — with that latency recorded in `latency_ms` (line 759) and included in the wall time used for `requests_per_sec` (line 845). No iterations are discarded as warmup anywhere in the file.

The delta report (`build_delta_report`, line 874) computes `rps_ratio_guest_over_host` and `pNN_delta_ms` directly from these warmup-contaminated numbers.

## Impact
Connection-establishment and task-spawn overhead is attributed to steady-state throughput/latency. This biases the abstraction-cost delta (the headline metric) — especially the p99 and `*_delta_ms` fields, where a single cold-connect outlier per worker shifts the tail. Regression detection thresholds built on these artifacts inherit the noise.

## Suggested fix
Add a warmup phase (issue and discard a small fixed number of requests per worker before starting `Instant::now()`), and capture `started` *after* the worker tasks are spawned and have reached their request loop (e.g. via a barrier). At minimum, document that cold-connection cost is included so consumers don't read the delta as steady-state.

## Triage
Confirmed from the local reviewed report in `/Users/elie/git/capsem/tmp/bugs/tooling-04-no-warmup-and-spawn-skew-in-throughput.md`. Track implementation in the triage sprint; add regression coverage before fixing.

Contributor guide

Open the contributing guide

Research direction

Start in crates/capsem-bench/src/main.rs at run_http_scenario (583-607) and run_dns_scenario (617-668), then inspect run_one_request and build_delta_report. Add regression coverage for warmup and timing boundaries; done means cold connection and worker-spawn overhead are excluded from steady-state latency and throughput metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.