Investigate post-restart RPC failure in ledger retention CI

@bzawisto is already working on this.

Since Sep 14, 2026.

## Outcome

Determine why `redshift/ledger_retention` lost RPC connectivity after a graceful validator restart in CI, identify the owning component, and produce a reproducible regression case or a concrete next-failure capture plan.

**The cause is unknown.** The evidence does not establish a RedSuite race, a stale HTTP connection, a validator crash, or an infrastructure failure. Filing this investigation in RedSuite is not attribution of the defect to RedSuite.

## Scope

### Original failure and preserved evidence

- [Failed job: RedSuite correctness, run 34837021434 / job 103953186996](https://github.com/magicblock-labs/magicblock-validator/actions/runs/34837021434/job/103953186996), 2026-09-14, push to MBV `dev`, attempt 1.
- Runner: `blacksmith-16vcpu-ubuntu-2404`. The workflow runs `redsuite run redshift --profile lite` and then Redhat. Redshift failed, so Redhat was not reached. The other 23 Redshift scenarios passed.
- [Previous RedSuite run 34833450673](https://github.com/magicblock-labs/magicblock-validator/actions/runs/34833450673) passed. The intervening MBV commit only updated `test-integration/Cargo.lock`; this does not identify the cause.

The decisive CI timeline, in UTC, is copied here so investigation does not depend on retained artifacts:

| Time | Observation |
| --- | --- |
| 11:14:17.216 | `starting redshift/ledger_retention` |
| 11:14:27.213 | Retention event 1: 1 pruned, 12 retained, 12 adds, 4 superblocks allocated |
| 11:14:34.591 | Retention event 2: 6 pruned, 26 retained, 31 adds, 7 superblocks allocated |
| 11:14:42.375 | Retention event 3: 11 pruned, 41 retained, 51 adds, 10 superblocks allocated |
| 11:14:42.436 | Old validator: `graceful shutdown has been requested` |
| 11:14:42.504 | Old validator: `process terminated cleanly`, `exit_code=0` |
| 11:14:42.516 | New validator starts |
| 11:14:42.559 | Ledger reopened with 7 superblocks |
| 11:14:42.683 | `reset volatile state`, `slot=391` |
| 11:14:42.785 | RPC server running; `Ready for connections!` |
| 11:14:42.793 | On-chain setup complete |
| 11:14:43.187 | Last new-boot log: `sequencer is halting operation` |
| 11:14:43.756 | RedSuite prints cleanup of private ER PID 5930, then reports the failure below |

```text
FAILED redshift/ledger_retention: scenario failed: http://127.0.0.1:35113: error sending request for url (http://127.0.0.1:35113/)
error: 1 of 24 scenarios failed
```

The report recorded `passed=false`, phase `scenario`, kind `infrastructure`, wall time `26.5367562` seconds, and one ER relaunch. The `infrastructure` classification is the harness's broad transport-error category, not proof of an external infrastructure fault. There was no recorded underlying socket error, failing RPC method, or new-process exit code/signal. The final sequencer message marks a normal quiescence/snapshot barrier; it is not itself evidence of shutdown or a crash. The old boot emitted the same message at earlier superblock boundaries.

Relevant Actions artifacts were `redsuite-stack-logs-correctness` (including current and previous retention boot logs), `redsuite-reports-correctness`, and `er-debug-bin`. Stack logs have a three-day retention policy; the essential observations are embedded above.

### Pinned inputs

| Component | Revision/version |
| --- | --- |
| MBV run revision | `54819a8fe77e7ee813bc7fa9c5857671d13e8155` |
| MBV cached binary source marker | `c83cae3`; the later run commit changes only the integration lockfile |
| Engine resolved from MBV lockfile | Tag `0.6.0`, commit `82c0048fff2313a2fe1a88b0d4b25c58227aab5d` |
| RedSuite | Tag `0.3.0`, commit `f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd` |
| RedSuite Rust toolchain | `1.94.1` |
| MBV Rust toolchain | `1.96.1` |
| Agave / solana-test-validator | `4.1.1` |

Exact CI binaries were used in the Linux attempts below. The artifact ZIP SHA-256 was checked against GitHub's recorded digest:

```text
er-debug-bin ZIP: 357c978d8d051a8a12a27e83cc2a79ab1e82324daf6a2b65c45668ebcd0ba67f
validator: 19fa336b06da7b6a0e2a36b6bad1e45d30d7fe70896d19388c932922e4bf96c1
verifier: 3df82d7ba51b17505eb54097b8266187684c258141ab1dc153a7310fc05293eb
```

### Scenario and source entry points

The [retention scenario](https://github.com/magicblock-labs/redsuite/blob/f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd/redshift/src/scenarios/storage/ledger_retention.rs) starts a private ER with 40-slot superblocks and a one-byte ledger size limit to force retention. It submits counter updates every 200 ms, waits for at least three truncations and observed history pruning (180-second budget), then checks history before and after a graceful restart and after one additional transaction. It verifies ordered pruning, retained transaction/block/address history, and counter state.

Investigate these boundaries without assuming which one failed:

- [PrivateEr::restart](https://github.com/magicblock-labs/redsuite/blob/f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd/crates/redsuite-core/src/topology/private.rs#L159-L207): reuses ports/storage/identity, probes primary readiness using a separate `Api`, preserves the scenario's existing context/client, and resets only its blockhash cache. Whether a preserved connection can cause this failure needs evidence; do not assume it does.
- [HTTP transport](https://github.com/magicblock-labs/redsuite/blob/f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd/crates/redsuite-core/src/transport/http.rs): stores only `reqwest::Error::to_string()`, losing the source chain.
- [RPC propagation](https://github.com/magicblock-labs/redsuite/blob/f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd/crates/redsuite-core/src/api.rs#L360-L370): does not attach the method to a transport error.
- [Private-ER drop](https://github.com/magicblock-labs/redsuite/blob/f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd/crates/redsuite-core/src/topology/private.rs#L212-L227): discards child wait status, obscuring whether the validator died before cleanup.
- [Engine pacemaker](https://github.com/magicblock-labs/magicblock-engine/blob/82c0048fff2313a2fe1a88b0d4b25c58227aab5d/engine/src/pacemaker.rs#L183-L204): superblock checkpoint/barrier path corresponding to the last validator message. A stack/core or other causal evidence is needed before attributing the failure here.

### Linux attempts already performed

Environment: Ubuntu 24.04.4, kernel `6.17.0-1022-gcp`, Intel Xeon Platinum 8581C, 24 vCPUs/47 GiB RAM; processes restricted to CPUs 0–15. AccountsDB and ledger used fresh isolated ext4 storage. Application listeners were loopback. FD limit was 1,048,576; inotify limits matched CI (1,280 instances / 655,360 watches). Public mainnet RPC supplied fixture clones; CI may use its secret-configured endpoint.

| Attempt | Result | Retention events | Final transactions tracked | Final pruned / retained | Shutdown / startup |
| --- | --- | --- | --- | --- | --- |
| Isolated `ledger_retention`, unchanged harness, process/signal tracing | Passed | 3 | 28 | 10 / 18 | 914 / 754 ms |
| Full concurrent Redshift suite, failure-only diagnostics, no tracing | 24/24 passed | 3 | 52 | 14 / 38 | 92 / 370 ms |

Both attempts passed post-restart history/state checks and the post-restart transaction. Shutdown is signal-to-exit; startup ends at primary-health readiness. Tracing materially changed timing, so traced durations are not performance measurements. The untraced suite's third retention event had 50 adds, close to CI's 51, but this does not prove equivalent scheduling. RedSuite was built with default release CPU flags; CI specifies native CPU flags. The exact validator/verifier binaries were unchanged. Two passing attempts do not exclude an intermittent defect.

### Reproduction setup without access to any existing test machine

Use an x86-64 Linux environment with the pinned toolchains, Agave 4.1.1 (`solana-test-validator` and `cargo-build-sbf` on PATH), and native build prerequisites from the pinned [MBV build workflow](https://github.com/magicblock-labs/magicblock-validator/blob/54819a8fe77e7ee813bc7fa9c5857671d13e8155/.github/workflows/build-er.yml) / setup actions. Use new checkouts and scratch directories, not valuable validator storage.

While retained, download the exact binaries with:

```sh
gh run download 34837021434 \
--repo magicblock-labs/magicblock-validator \
--name er-debug-bin --dir er-artifact
chmod +x er-artifact/debug/magicblock-validator er-artifact/debug/magicblock-verifier
```

If the artifact expires, build from the pinned source in a fresh MBV checkout; this is a source-equivalent attempt, not a byte-identical artifact:

```sh
git checkout --detach 54819a8fe77e7ee813bc7fa9c5857671d13e8155
CARGO_PROFILE_DEV_DEBUG=line-tables-only \
RUSTFLAGS='-C link-arg=-fuse-ld=lld' \
cargo build --locked --bin magicblock-validator --bin magicblock-verifier
cargo build-sbf --manifest-path magicblock-committor-program/Cargo.toml \
--sbf-out-dir target/deploy/ --quiet
```

In a fresh RedSuite checkout:

```sh
git checkout --detach f0ac45ec9ad4eaa2468147cc38b74b9ddcb3cacd
cargo build --release --locked -p redsuite
cargo xtask programs

# Absolute paths to the downloaded artifact, or to MBV target/debug above.
export MAGICBLOCK_VALIDATOR_BIN=/absolute/path/to/er-artifact/debug/magicblock-validator
export MAGICBLOCK_VERIFIER_BIN=/absolute/path/to/er-artifact/debug/magicblock-verifier
# Keep deploy/magicblock_committor_program.so beside the debug/ directory.
export RUST_BACKTRACE=1
ulimit -n 1048576
unset REDSUITE_ROOT

target/release/redsuite run redshift/ledger_retention --profile lite
target/release/redsuite stack down

# Also exercise CI's shared/private scenario concurrency.
target/release/redsuite run redshift --profile lite
target/release/redsuite stack down
```

On a machine with CPUs 0–15 available, prefix the run commands with `taskset -c 0-15` to match the attempts above. Retain each attempt's reports and current/previous boot logs before another run overwrites them. Ensure stack cleanup also runs on scenario failure. RedSuite uses public mainnet for cloning by default; record any endpoint substitution without publishing credentials. Repository-relative outputs are `target/redsuite-reports` and `target/redsuite-stack`.

### Capture requirements for the next failure

The second attempt used only these temporary diagnostics, which anyone can reproduce in an isolated checkout:

1. In `TransportError::request`, replace `detail: error.to_string()` with `detail: format!("{error:?}")` to expose the reqwest/hyper/OS cause.
2. At `Api::call_nullable`'s `post_json(...).await`, log the method on error and return the **same error**, preserving downcast/classification behavior.
3. In private-ER cleanup, record `child.try_wait()` **before** sending cleanup signals and record the final `child.wait()` result. Distinguish an earlier fatal signal from normal harness termination.
4. Add operation labels around post-restart `observe`, `verify`, and `send_add` when narrowing the failed call; the original log does not identify which of these failed.

No retry policy, assertions, readiness condition, scenario workload, or validator code was changed in those attempts. Expected connection refusals during readiness polling are not failures. Additional tracing can use `strace -f -tt -e trace=process -e signal=all`; it changes timing and stays attached to the detached base validator until explicit stack cleanup. Capture a core/backtrace if a fatal signal occurs, and kernel/cgroup memory events if resource termination is suspected. Do not infer a crash from a truncated log alone.

Related work: #125 tracks durable error/child-status reporting; #126 tracks the separately confirmed macOS process-supervision defect. Neither proves the cause of this CI failure. They are related issues, not prerequisites for using temporary Linux diagnostics. If the cause belongs to MBV or Engine, link and transfer/file the concrete fix in the owning repository.

## Acceptance criteria

- [ ] A failing attempt records exact revisions, build mode, environment, scenario concurrency, operation/RPC method, full transport cause, child status before cleanup, and correlated boot logs. Publish decisive evidence in the issue or a durable repository-accessible artifact, not a private machine path.
- [ ] Establish whether the cause is harness/client lifecycle, validator behavior, or environment using causal evidence. If it remains unreproduced, document the bounded attempts and enable a specific next-failure capture path rather than declaring it a flake or closing on passing reruns.
- [ ] For a confirmed defect, provide a minimal reproducer/regression case and an owning-component fix or linked follow-up with clear verification criteria.
- [ ] Preserve ordered retention, post-restart state/history integrity, signer/synchronization checks, resource bounds, and critical-path performance. Do not hide the symptom with blanket retries, longer timeouts, weaker assertions, or disabled coverage.
- [ ] Verification distinguishes an isolated scenario pass from concurrent-suite results and records whether tracing or diagnostics altered timing.