google / google/capsem

HTTP MITM path has severe fixed per-request overhead

Open
#96 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

## Problem

Capsem's guest-through-MITM HTTP path has a serious fixed per-request overhead. The local mock server itself is fast, but tiny HTTP requests collapse when routed through Capsem.

This is release-blocking or near release-blocking for agent responsiveness because small HTTP/model/control requests are common and the latency shape matches manual TUI/UI lag.

## Measured data

Command shape:

```bash
capsem-bench-rs protocol-delta \
--base-url http://127.0.0.1:3713 \
--guest-base-url http://127.0.0.1:3713 \
--session code-bench-1 \
--scenarios tiny_http

capsem-bench-rs protocol-delta \
--base-url http://127.0.0.1:3713 \
--guest-base-url http://127.0.0.1:3713 \
--session code-bench-1 \
--scenarios http_10mb
```

Results:

| Scenario | Host direct | Guest through Capsem | Delta |
| --- | ---: | ---: | ---: |
| `tiny_http`, 50k requests, c=64 | `113,284 rps`, p50 `0.5ms`, p95 `1.0ms`, p99 `1.5ms` | `4,440 rps`, p50 `7.1ms`, p95 `32.3ms`, p99 `40.0ms` | `25.5x` slower, `+31.3ms` p95 |
| `http_10mb`, 512 requests, c=16 | `7.31 GB/s`, p50 `22.1ms`, p95 `27.5ms`, p99 `44.6ms` | `1.52 GB/s`, p50 `108.5ms`, p95 `116.9ms`, p99 `142.1ms` | `4.8x` slower, `+89.4ms` p95 |

Artifacts from the run were written locally to:

- `/tmp/capsem-bench-tiny-delta.json`
- `/tmp/capsem-bench-download-delta.json`

## What this means

The mock server is not the bottleneck. Host direct can do `113k rps` for tiny HTTP and `7.31 GB/s` for 10MB responses.

The worst issue is fixed per-request overhead. Tiny HTTP collapses much harder than 10MB download, so the first target should be per-request MITM work, not raw byte forwarding.

Likely suspects:

- response-end telemetry work in the MITM hook
- plugin/security rail execution per HTTP event
- body hook dispatch overhead per response
- per-request task spawning or lock contention around ledger/security handoff
- excessive metrics/logging on the hot path

## Required next diagnostic

Run the same tiny HTTP benchmark with the newly-added counters scraped or logged, and identify the dominant stage before changing architecture.

Counters to inspect:

- `mitm.telemetry_response_end_duration_ms`
- `mitm.telemetry_stage_duration_ms{stage=...}`
- `security_plugin.execution_duration_ms{plugin_id,stage,mode,...}`
- `db.enqueue_wait_ms`
- `db.enqueue_lock_wait_ms`
- `db.producer_batch_send_ms`
- `db.producer_buffer_size`

The fix should be backed by:

1. A before/after `capsem-bench-rs protocol-delta` table for `tiny_http` and `http_10mb`.
2. A regression gate for tiny HTTP guest-through-Capsem p95/rps.
3. A proof that large download throughput does not regress.
4. No Python load generator for HTTP release truth; use `capsem-bench-rs`.

Contributor guide

Open the contributing guide

Research direction

Start by running the two capsem-bench-rs protocol-delta scenarios and scrape or log the listed MITM, security-plugin, and database counters. Identify the dominant per-request stage before changing architecture. Done means a before/after table for tiny_http and http_10mb, a tiny-HTTP regression gate, and proof that large-download throughput does not regress.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.