Build a black-box validator test harness

@bzawisto is already working on this.

Since Jun 30, 2026.

## Goal

Build a black-box validator test harness.

## Why

The current behavior or architecture does not provide this outcome.

## Scope

**Problem**
Our unit and integration tests cover a lot of correctness paths, but they don't reproduce the failures that only show up when the whole validator is under sustained, realistic load. At runtime, RPC ingress, account sync, scheduling and execution, AccountsDb and ledger persistence, committor settlement, task scheduling, replication, and metrics all interact, and most of the regressions we've hit on devnet came out of those interactions — usually around startup/shutdown, replication, upstream provider pressure, cloning and subscriptions, or restart readiness.

`test-integration` has also become hard to maintain. It's a large pile of crates and cases with a custom runner, per-suite configs, a shared context type, and several throwaway programs, each suite doing setup and assertions its own way. Coverage is uneven, and it mixes black-box checks over RPC/WS with white-box checks that reach into internal mocks.

So devnet is usually where we first notice a release regression. We'd rather catch those locally, before deploying, by running a release candidate under realistic load and comparing it against the last released version.

**Proposing a single black-box harness: "RedSuite"**
This issue sketches one test harness — working name **RedSuite** — covering both the release-load/regression testing described here and the correctness testing we get today from `test-integration`.

RedSuite would be a standalone project, in its own repo, that treats the validator, protocol, and programs as a black box: it interacts only through the public surface — transactions, JSON-RPC, WebSocket, and the Prometheus metrics endpoint — and never touches internal crates. It aims to cover that whole surface, and to exercise the happy path, accidental misuse, and deliberate abuse.

It would gradually take over from `test-integration` for everything checkable from outside. The checks that genuinely need internal access (the `test-chainlink` mocks, say) move to unit tests in their own crates. The split: observable from outside → RedSuite, on release/nightly as an advisory signal; only visible inside → a unit test in its home crate, per-PR. The release-load/regression report below is then one mode of RedSuite, not a separate tool.

**Building on redline**
`magicblock-labs/redline` is our load-testing tool, used for ad-hoc benchmarking. It supports TPS/RPS, mixed workloads, configurable concurrency, clone cadence, commit mode, and subscriptions, with run comparison. It's a solid base for RedSuite and brings a lot of reusable plumbing: streaming stats, HTTP/WS pooling, confirmation tracking, rate limiting, an account-prep CLI, and a small on-chain program with load workloads.

RedSuite would grow out of redline rather than start fresh, pulling its load program and workloads in as the load layer. Once RedSuite covers what redline does, the standalone tool and the `redline.so` preloaded in two integration configs can be retired.

**What we'd build**
A few layers — the lower ones largely from redline, the upper ones new:

- **The harness** — stands up and tears down an isolated topology (local base layer, the ephemeral validator under test, optional primary/replica for replication releases), handles config and account prep, and can start, kill, restart from an existing ledger, and clean up. The restart support is what lets us cover ledger-restore and recovery from the outside.
- **A public-API client** — one client over the JSON-RPC methods, WS subscriptions, and transaction submission a real user has access to, replacing the per-suite context type.
- **Test programs** — a small set of on-chain programs to drive behavior, grouped by purpose: `redhat` - security/adversarial (wrong signer or owner, malformed or oversized inputs, interrupted clones, replay), `redshift` - correctness (exact end-state across the delegate → commit → undelegate lifecycle, schedule-commit and intents, restarts), and `redline` - load (stress workloads plus consistency-under-contention checks).
- **Scenarios** — each sets up, drives the system through the client, checks expectations, tears down, and reports violations and measurements.
- **A generative layer, later** — once the hand-written scenarios exist, a property/fuzz layer that runs valid and invalid sequences against a model of expected state and shrinks any failure to a minimal reproducer.

**Replacing test-integration**
RedSuite would absorb what `test-integration` covers today, one area at a time — cloning, delegation, committor, schedule-commit and intents, pubsub, ledger restore, task scheduling, address-lookup tables, config — re-checking each through the public API rather than internal state. The chainlink suite is the main exception: the part that asserts on internal mocks moves to unit tests in `magicblock-chainlink`. The target is that every public RPC method, WS subscription, and magic-program instruction is exercised by at least one scenario.

**Local harness scope**
The first version runs locally with an isolated topology, not against public devnet, and tears everything down after each run — no devnet downtime and no dependence on shared devnet nodes for the release signal. An optional primary/replica topology covers replication-enabled releases.

**Workload and metrics**
The release run should push the validator to capacity with a realistic mix: RPC reads, parallel and contended transactions, high-CU and mixed read/write, account cloning and subscriptions, commit-heavy settlement, and the startup/restart and replication paths where they apply. Private/TEE auth is in scope where a release needs it.

Each run collects both load-side numbers (throughput, latency percentiles, error and timeout counts) and validator-side metrics (transaction/clone/commit counts and failures, backlog and pending-clone gauges, upstream RPC pressure, pubsub stream health, startup/shutdown timing, resource use). Gaps in instrumentation become follow-up issues.

**Regression report**
Non-blocking to start: it shows whether a candidate improved, regressed, or stayed flat against the previous release under the same load — correctness failures, crashes, unrecovered failures, growing backlog, latency and throughput movement, RPC amplification, restart timing, resource use. Thresholds get calibrated from repeated runs before any of it becomes a hard gate.

**Rough order of work**
1. Start from redline; refactor its core, transport, and prep into the new crates.
2. Grow the load program from redline's, add the security and correctness programs.
3. Port areas one at a time; delete each `test-integration` crate once its replacement passes against `master`.
4. Add the generative layer once hand-written coverage is solid.
5. Retire standalone redline and drop the `redline.so` preloads.
6. Delete `test-integration` and wire RedSuite into release pipeline.

**References**
- `magicblock-labs/redline` (the prototype this would grow from), and `magicblock-labs/redline#6` (TEE auth)
- `magicblock-labs/magicblock-validator#445`
- https://github.com/magicblock-labs/magicblock-validator-review-report