HTTP MITM path has severe fixed per-request overhead
- Dominant language
- Rust
- Stars
- 72
- Forks
- 13
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 5
Description
## Problem
Capsem's guest-through-MITM HTTP path has a serious fixed per-request overhead. The local mock server itself is fast, but tiny HTTP requests collapse when routed through Capsem.
This is release-blocking or near release-blocking for agent responsiveness because small HTTP/model/control requests are common and the latency shape matches manual TUI/UI lag.
## Measured data
Command shape:
```bash
capsem-bench-rs protocol-delta \
--base-url http://127.0.0.1:3713 \
--guest-base-url http://127.0.0.1:3713 \
--session code-bench-1 \
--scenarios tiny_http
capsem-bench-rs protocol-delta \
--base-url http://127.0.0.1:3713 \
--guest-base-url http://127.0.0.1:3713 \
--session code-bench-1 \
--scenarios http_10mb
```
Results:
| Scenario | Host direct | Guest through Capsem | Delta |
| --- | ---: | ---: | ---: |
| `tiny_http`, 50k requests, c=64 | `113,284 rps`, p50 `0.5ms`, p95 `1.0ms`, p99 `1.5ms` | `4,440 rps`, p50 `7.1ms`, p95 `32.3ms`, p99 `40.0ms` | `25.5x` slower, `+31.3ms` p95 |
| `http_10mb`, 512 requests, c=16 | `7.31 GB/s`, p50 `22.1ms`, p95 `27.5ms`, p99 `44.6ms` | `1.52 GB/s`, p50 `108.5ms`, p95 `116.9ms`, p99 `142.1ms` | `4.8x` slower, `+89.4ms` p95 |
Artifacts from the run were written locally to:
- `/tmp/capsem-bench-tiny-delta.json`
- `/tmp/capsem-bench-download-delta.json`
## What this means
The mock server is not the bottleneck. Host direct can do `113k rps` for tiny HTTP and `7.31 GB/s` for 10MB responses.
The worst issue is fixed per-request overhead. Tiny HTTP collapses much harder than 10MB download, so the first target should be per-request MITM work, not raw byte forwarding.
Likely suspects:
- response-end telemetry work in the MITM hook
- plugin/security rail execution per HTTP event
- body hook dispatch overhead per response
- per-request task spawning or lock contention around ledger/security handoff
- excessive metrics/logging on the hot path
## Required next diagnostic
Run the same tiny HTTP benchmark with the newly-added counters scraped or logged, and identify the dominant stage before changing architecture.
Counters to inspect:
- `mitm.telemetry_response_end_duration_ms`
- `mitm.telemetry_stage_duration_ms{stage=...}`
- `security_plugin.execution_duration_ms{plugin_id,stage,mode,...}`
- `db.enqueue_wait_ms`
- `db.enqueue_lock_wait_ms`
- `db.producer_batch_send_ms`
- `db.producer_buffer_size`
The fix should be backed by:
1. A before/after `capsem-bench-rs protocol-delta` table for `tiny_http` and `http_10mb`.
2. A regression gate for tiny HTTP guest-through-Capsem p95/rps.
3. A proof that large download throughput does not regress.
4. No Python load generator for HTTP release truth; use `capsem-bench-rs`.
Contributor guide
Research direction
Start by running the two capsem-bench-rs protocol-delta scenarios and scrape or log the listed MITM, security-plugin, and database counters. Identify the dominant per-request stage before changing architecture. Done means a before/after table for tiny_http and http_10mb, a tiny-HTTP regression gate, and proof that large-download throughput does not regress.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100