google / google/capsem

Investigate per-profile exec latency: co-work boots ~20% slower than code

Open
#173 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

## Observation

Provision-to-first-exec latency differs materially by profile. Measured on Apple Silicon during the 0.6 release lane:

| profile | exec latency |
|---|---|
| `code` | comfortably under 1.5s |
| `co-work` | **1.78s / 1.79s** (stable across runs) |

The `_test-functional` stage runs a profile matrix (`code co-work`) and asserts the same latency budget for both. `code` passes, `co-work` fails — reproducibly, not intermittently. Two consecutive lane runs measured 1.78s and 1.79s.

This blocked the 0.6 release lane and initially read as a flake, because only the *second* serial invocation (the co-work leg) fails while the first (`code`) passes in the same run.

## Immediate action taken

`EXEC_LATENCY_GATE` raised to 2.0s for all platforms in `tests/capsem-serial/test_boot_timing.py` (Linux was already 2.0s; macOS was 1.5s). That unblocks the lane but does not explain the difference.

## What to investigate

1. **Where the time goes.** `process.log` carries `boot_timeline` spans — compare the two profiles span by span rather than assuming it is image size.
2. **Whether this is a regression or inherent.** `target/test-benchmarks/` holds recorded history. Has `co-work` ever been under 1.5s? If it regressed, raising the gate hid a real problem and this should be reverted rather than accepted.
3. **Image size and composition.** `co-work` carries more packages than `code`. Quantify the rootfs delta and whether package count actually drives first-exec latency, or whether it is something else (larger initrd, more services started at boot, slower overlay setup).
4. **Optimization candidates**, if the cost is inherent:
- rootfs compression choice and decompression cost at boot
- trimming or deferring packages not needed before first exec
- lazy service start so first-exec does not wait on the full boot sequence
- whether VirtioFS vs block mode changes the picture

## Why it matters

Exec latency is the number users feel first. A profile being 20% slower to reach usable state is a product characteristic worth knowing deliberately rather than discovering through a red gate — and if a heavier profile inherently costs more, the budget should probably be per-profile rather than a single global number.

## Pointers

- `tests/capsem-serial/test_boot_timing.py` — the gate
- `justfile` — functional profile matrix (`code co-work`)
- `config/profiles//` — profile ledgers
- `/dev-benchmark` and `/dev-testing` skills

Contributor guide

Open the contributing guide

Research direction

Start with the functional profile matrix in justfile and the gate in tests/capsem-serial/test_boot_timing.py, then compare process.log boot_timeline spans for code and co-work. Review target/test-benchmarks/ and the ledgers under config/profiles//, and quantify the image differences. Done means identifying where the latency gap comes from and establishing whether the raised gate should be reverted or made profile-specific.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems, performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.