[triage:tooling-04-no-warmup-and-spawn-skew-in-throughput] Throughput timing includes worker-spawn skew and has no warmup; first-request connection setup is counted
- Dominant language
- Rust
- Stars
- 72
- Forks
- 13
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 5
Description
Imported from Capsem triage report `tooling-04-no-warmup-and-spawn-skew-in-throughput.md`.
- Severity: `low`
- Category: `bug (measurement correctness)`
- Area: `capsem-bench`
- Location: `crates/capsem-bench/src/main.rs:583-607` (`run_http_scenario`), `617-668` (`run_dns_scenario`)
- Confidence: `verified`
## Summary
Each scenario's wall clock starts *before* the workers are spawned and stops after all join. There is no warmup phase, so TCP connect + TLS handshake for the first request on each pooled connection, plus tokio task-spawn scheduling latency, is folded into both per-request latency samples and the scenario-level `requests_per_sec` / `bytes_per_sec`. For low request counts or high-latency lanes (the guest-through-Capsem lane), cold-connection cost measurably distorts the host-vs-guest delta the tool exists to compute.
## Evidence
`run_http_scenario` (lines 583-599):
```rust
let started = Instant::now();
let tasks = (0..workers).map(|idx| { ... tokio::spawn(async move { ... }) });
let joined = try_join_all(tasks).await...;
let wall_time = started.elapsed();
```
`started` is captured before the `tokio::spawn` closures are constructed and scheduled; `wall_time` therefore includes spawn/scheduling skew across `workers` tasks. The first `run_one_request` per worker (line 591) performs `request.send()` (line 754) on a fresh pooled connection — connect + TLS for HTTPS lanes — with that latency recorded in `latency_ms` (line 759) and included in the wall time used for `requests_per_sec` (line 845). No iterations are discarded as warmup anywhere in the file.
The delta report (`build_delta_report`, line 874) computes `rps_ratio_guest_over_host` and `pNN_delta_ms` directly from these warmup-contaminated numbers.
## Impact
Connection-establishment and task-spawn overhead is attributed to steady-state throughput/latency. This biases the abstraction-cost delta (the headline metric) — especially the p99 and `*_delta_ms` fields, where a single cold-connect outlier per worker shifts the tail. Regression detection thresholds built on these artifacts inherit the noise.
## Suggested fix
Add a warmup phase (issue and discard a small fixed number of requests per worker before starting `Instant::now()`), and capture `started` *after* the worker tasks are spawned and have reached their request loop (e.g. via a barrier). At minimum, document that cold-connection cost is included so consumers don't read the delta as steady-state.
## Triage
Confirmed from the local reviewed report in `/Users/elie/git/capsem/tmp/bugs/tooling-04-no-warmup-and-spawn-skew-in-throughput.md`. Track implementation in the triage sprint; add regression coverage before fixing.
Contributor guide
Research direction
Start in crates/capsem-bench/src/main.rs at run_http_scenario (583-607) and run_dns_scenario (617-668), then inspect run_one_request and build_delta_report. Add regression coverage for warmup and timing boundaries; done means cold connection and worker-spawn overhead are excluded from steady-state latency and throughput metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100