google / google/capsem

Port remaining capsem-bench legacy modes to Rust and retire Python benchmark wrapper

Open
#95 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

## Problem

`capsem-bench protocol` must be Rust-backed so HTTP/model/credential/MCP/DNS hot-path benchmarks use the same implementation in host and VM lanes. The current release fix makes the Rust protocol binary mandatory in the guest as `capsem-bench-rs` and makes the existing `capsem-bench` wrapper delegate `protocol` to it.

The remaining `capsem-bench` modes are still implemented by the legacy Python package:

- `disk`
- `rootfs`
- `storage`
- `startup`
- `snapshot`
- `all`
- legacy load modes

This keeps release compatibility, but it is not the desired end state.

## Required end state

- One Rust `capsem-bench` implementation owns all benchmark modes.
- No Python benchmark wrapper is needed in the VM image.
- Host and VM lanes use the same Rust code for all hot protocol and release benchmarks.
- `capsem-bench all` remains available and writes `/tmp/capsem-benchmark.json` with the existing release artifact contract or an explicitly migrated schema.
- Existing benchmark gates and docs are updated to the Rust implementation.

## Gates

- Rust unit tests cover every mode.
- In-VM `capsem-bench all` passes and archives the expected JSON artifact.
- Protocol delta can run HTTP, model, credential, MCP/tools, and DNS in both host and guest lanes.
- No `guest/artifacts/capsem_bench` Python package remains after the port.
- Docker/rootfs/initrd packaging tests prove the Rust binary is the only benchmark executable shipped.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Rust `capsem-bench` implementation and the existing `capsem-bench` wrapper across the listed modes: `disk`, `rootfs`, `storage`, `startup`, `snapshot`, `all`, and legacy load modes. Review the Rust unit tests, in-VM `capsem-bench all` gate, and Docker/rootfs/initrd packaging tests. Done means all modes use Rust, the expected `/tmp/capsem-benchmark.json` artifact is archived, protocol lanes pass, and `guest/artifacts/capsem_bench` is gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, rust
Domain
build-system, devtools, release, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.