Automattic / Automattic/wp-codebox

Build a first-class adversarial WordPress fuzzing and fault-injection runtime

Open
#2,014 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

## Problem

WP Codebox has excellent disposable WordPress runtimes, deterministic recipes, browser observation, multi-actor request gates, MySQL provisioning, checkpoints, replay packages, and artifact capture. Those primitives support authored fuzz campaigns, but they do not yet form a first-class adversarial fuzzing runtime capable of systematically finding unknown security, correctness, concurrency, and user-facing defects in arbitrary plugins, themes, or WordPress core.

Today a campaign author must supply most mutation logic, state exploration, service failure behavior, security oracles, corpus management, and minimization. Browser routing can allow/block/record traffic, but there is no transport-neutral fault model. `fuzzRun` records case/replay metadata, but it is not a coverage-guided mutation engine. MySQL is the only built-in disposable service, and its lifecycle cannot be disrupted mid-run.

The goal is not to promise that every zero-day will be found. The goal is to make WP Codebox capable of sustained, deterministic, hacker-grade adversarial testing that can aggressively attack a component's assumptions while preserving a safe, minimized, artifact-backed reproduction for every finding.

## Existing substrate to extend

- `packages/runtime-core/src/recipe-schema.ts`: `fuzzRun`, `distribution.serviceFakes`, overlays, services, artifacts, replay metadata.
- `packages/runtime-playground/src/browser-preview-routing.ts`: browser host allow/block/routing.
- `packages/runtime-playground/src/browser-multi-actor-scenario*.ts`: seeded actors, barriers, held requests, and deterministic schedules.
- `packages/runtime-playground/src/browser-capture-session.ts`: console, page errors, network, waterfall, DOM, screenshots, and diagnostics.
- `packages/cli/src/runtime-services.ts`: disposable MySQL/MariaDB service lifecycle.
- `packages/runtime-playground/src/runtime-snapshot.ts` and `replayable-wordpress-site-bundle.ts`: checkpoints and WordPress state replay.
- Typed artifact manifests, runtime attestations, mounted-file diffs, and sealed evidence bundles.

These should be composed, not replaced.

## Missing primitives

### 1. Stateful, coverage-guided mutation engine

Provide a generic campaign runner that can:

- maintain seed corpora and interesting-case queues;
- mutate scalar, structured, binary, multipart, JSON, XML, serialized PHP, block markup, shortcode, HTML, URL, header, cookie, and filesystem inputs;
- generate stateful action sequences rather than independent requests;
- use PHP/JavaScript/request/route/state coverage or novelty signals to retain cases;
- cross role, capability, multisite membership, authentication, locale, PHP, WordPress, and component-version matrices;
- enforce per-case resource budgets;
- deterministically replay a seed and normalized schedule;
- automatically shrink inputs and action sequences to a minimal reproduction.

WordPress-specific grammars belong in the WordPress extension. Core runtime mutation, scheduling, corpus, and minimization must remain generic.

### 2. Declarative transport fault model

Add typed request matching and scripted outcomes for browser and server-side traffic:

```json
{
"match": {
"host": "service.example",
"method": "POST",
"path": "/verify"
},
"sequence": [
{ "status": 500 },
{ "delayMs": 30000 },
{ "status": 200, "body": "malformed" }
]
}
```

Required fault classes:

- status/header/body substitution;
- malformed/truncated/chunked responses;
- arbitrary delay, jitter, bandwidth, and timeout stages;
- connection refusal, reset, half-close, and disconnect-after-N-bytes;
- DNS/host remapping;
- request/response corruption;
- deterministic response sequences and stateful counters;
- full request/fault/response evidence with secret redaction.

The same fault definition should work wherever technically possible across browser requests and WordPress HTTP transports, while reporting unsupported transport semantics explicitly.

### 3. Disruptable service environments

Extend `inputs.services` beyond MySQL and expose safe mid-run controls:

- Redis;
- SMTP/message sink;
- generic HTTP fixture service;
- optional object-storage/search fixtures where declared by an extension.

Fault controls should include stop, pause, restart, disconnect, latency, connection exhaustion, read-only mode, eviction/flush, malformed replies, disk-full/resource exhaustion, and recovery. MySQL needs mid-run disruption rather than provision-only lifecycle.

### 4. Server clock and scheduler control

Provide deterministic freeze, advance, skew, and restore controls for:

- PHP/WordPress time;
- browser time;
- cron/scheduled events;
- database comparisons where feasible.

This is required for expiry, nonce, token, cache, lock, retry, scheduling, and replay testing.

### 5. Adversarial browser and UI exploration

Build on browser scenarios to support:

- DOM-derived action/state graph exploration;
- generated click/fill/submit/navigation/back/reload sequences;
- rapid repeat, double-submit, stale-tab, multi-tab, and multi-user interactions;
- desktop/mobile/slow-device/locale/timezone/content-size matrices;
- malformed, empty, oversized, Unicode, bidi, and hostile rich-content inputs;
- layout oracles for overflow, overlap, clipping, invisible focus, dead controls, and stuck loading states;
- accessibility scanning;
- screenshot/DOM semantic regression comparison;
- automatic minimization to the shortest failing user journey.

Real browser engines and assistive technologies should be extension/provider capabilities, not assumptions in the generic runtime.

### 6. Security and integrity oracles

Campaigns need reusable, composable assertions for:

- authentication and authorization invariants;
- cross-user, cross-role, cross-site, and tenant isolation;
- nonce/CSRF and replay expectations;
- secret/PII leakage in responses, URLs, logs, HTML, screenshots, artifacts, and database state;
- unexpected privilege or session persistence;
- stored/reflected DOM execution indicators;
- filesystem escape and unexpected file mutation;
- database invariant and transactional consistency checks;
- duplicate side effects and idempotency;
- fail-open behavior after dependency/runtime faults;
- fatal, warning, notice, 5xx, unhandled rejection, console error, deadlock, livelock, timeout, and resource-growth detection.

The runtime should provide generic oracle plumbing; WordPress-specific policies should be registered by the WordPress extension or campaign.

### 7. Runtime instrumentation

Expose deterministic evidence useful for discovering novel paths:

- PHP and JavaScript coverage when available;
- WordPress hook, REST route, ability, AJAX action, cron, block render, CLI, and XML-RPC coverage via extension instrumentation;
- database query and mutation traces;
- filesystem mutation traces;
- cache operations;
- process memory, CPU, request duration, and leak trends;
- lock waits and concurrency schedules;
- sanitizers or engine diagnostics where supported by the backend.

Instrumentation must be opt-in, bounded, and artifact-safe.

### 8. Differential and compatibility fuzzing

Run the same minimized corpus across declared matrices:

- PHP versions;
- WordPress releases/trunk;
- component base versus candidate;
- SQLite versus MySQL/MariaDB;
- cache/service enabled versus disabled/faulted;
- browser/device profiles.

Classify findings as candidate regressions, pre-existing defects, platform differences, or nondeterminism.

### 9. First-class triage and replay

Every finding should produce a sealed reproduction containing:

- seed and corpus identity;
- minimized input and action schedule;
- exact component/runtime provenance;
- fault schedule;
- checkpoint or replayable WordPress state;
- relevant request, browser, database, filesystem, console, and runtime evidence;
- secret scan/redaction report;
- one-command replay instruction;
- stable finding fingerprint for deduplication.

A failed fuzz run without a replayable minimized case should be considered incomplete evidence.

## Safety requirements

- Network denied by default; external access must be declared and recorded.
- No host production credentials or state in a fuzz runtime.
- Bounded CPU, memory, disk, wall time, process count, and artifact size.
- Service and worker teardown must be guaranteed after crashes/timeouts.
- Secrets and personal data must be redacted before artifact publication.
- Dangerous payloads remain inside the disposable runtime and cannot address arbitrary host paths or services.
- Generic WP Codebox layers must not contain WordPress plugin, vendor, or transport-specific special cases; those belong in registered extension configuration.

## Proposed delivery phases

1. **Deterministic engine:** corpora, structured mutation, stateful sequences, novelty signals, shrinking, replay, and finding fingerprints.
2. **HTTP faults:** typed matchers, response sequences, latency/timeouts/resets, WordPress HTTP adapter, and browser adapter.
3. **Service faults:** Redis, SMTP, HTTP fixtures, and disruptable MySQL lifecycle.
4. **UI adversary:** DOM state exploration, layout/accessibility oracles, visual evidence, and journey minimization.
5. **Differential matrices:** component/runtime comparisons and regression classification.
6. **WordPress attack packs:** extension-owned grammars and oracles for REST, Abilities, AJAX, XML-RPC, blocks, options/meta, filesystem, cron, CLI, roles/capabilities, and multisite isolation.

## Acceptance criteria

- A recipe can run a deterministic adversarial campaign against an arbitrary mounted component without campaign-specific host scripting.
- A seeded intentionally vulnerable plugin/theme fixture proves discovery and minimization of authorization, injection, state corruption, race, external-service failure, and UI-state defects.
- At least one campaign demonstrates true concurrent workers plus deterministic schedule replay.
- At least one campaign demonstrates Redis, SMTP, HTTP, and MySQL fault/recovery behavior.
- At least one browser campaign automatically discovers and minimizes a user-facing interaction failure.
- Differential runs classify base/candidate and runtime-version behavior.
- Every retained finding has a stable fingerprint, bounded artifact bundle, secret scan, exact provenance, and one-command replay.
- Existing recipe, browser, service, artifact, and checkpoint contracts remain composable and backward compatible.

## Non-goals

- Claiming complete vulnerability detection or replacing expert security review.
- Baking one scanner, plugin, vendor, or WordPress-specific attack policy into the generic runtime.
- Allowing fuzz workloads to escape the Codebox sandbox or contact undeclared production services.
- Requiring every backend to support transport faults it cannot faithfully emulate; capability negotiation and explicit unsupported results are preferable to fake fidelity.

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin by reading packages/runtime-core/src/recipe-schema.ts, packages/runtime-playground/src/browser-preview-routing.ts, browser-multi-actor-scenario*.ts, runtime-snapshot.ts, and packages/cli/src/runtime-services.ts. Start by scoping the deterministic engine phase and identifying the existing fuzzRun, replay, artifact, and service contracts it must compose with. Done requires a narrowly defined, tested phase with deterministic replay and backward-compatible contracts.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mysql, php, redis, typescript
Domain
backend, databases, devtools, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.