Automattic / Automattic/wp-codebox

Benchmark aggregation reports skipped workloads as successful timing samples

Open
#2,495 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

## Observed failure
The MDI decision matrix completed with PASS for both backends, including a boot-timing workload that returned `skipped: missing_bench_boot_phase`. Aggregation reported approximately 0.004 ms as its duration, presenting the skip path as a benchmark result rather than a skipped scenario.

Reproduction workload: Automattic/markdown-database-integration, `tests/bench/boot-timing.php`, matrix source `21267e18d41c88396e4883e227422158ba5c16b2`. Invoke the decision benchmark profile without BENCH_BOOT_PHASE. Operator evidence: Homeboy job `7018d7ce-fdf0-442d-a95c-ad96eedac2c5`, SQLite run `b28d11dd-e33d-40cf-8944-683aacb5fbc3`, native run `2cc3b041-9291-4e8b-823f-e05b92376607`. The source and normal workload invocation are the portable reproduction; IDs require operator access.

## Expected contract
The workload result boundary preserves typed skip status and reason. Skipped workloads contribute no performance sample and are explicitly represented in matrix completeness. A required skipped scenario leaves acceptance incomplete, even when other executed scenarios pass. Test aggregation with executed, skipped, and failed callables through the real dispatcher.

Inspect owning layers before implementation: if WP Codebox already preserves skip status and the loss occurs downstream, route the repair to that aggregator rather than duplicating logic.

## AI Assistance
GPT-6 Astra through OpenCode inspected the benchmark output and workload source, searched existing trackers, and drafted this report under Chris Huber’s direction. Direct OpenCode repair runs use openai/gpt-5.6-terra.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with tests/bench/boot-timing.php through the decision benchmark profile without BENCH_BOOT_PHASE, then trace the result through the real dispatcher and the owning aggregation layer. Exercise executed, skipped, and failed callables and verify that skipped workloads produce no timing sample, retain their reason, and leave required matrix completeness incomplete.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, typescript, wordpress
Domain
backend, performance, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.