Flaky go-tests-short shard 4: runner dies mid-job, no test-level signal

## Summary

`go-tests-short` parallel runner index **4** has failed at least three times in close succession with the same symptom pattern. Same shard, same package set, same final visible log line — but different durations and at least two distinct underlying causes (OOM and "something else"). The streamed CI log truncates at a buffered package boundary that gives no hint as to which test was running at kill time, and the gotestsum jsonfile / JUnit XML for the failed shard is unrecoverable because `store_artifacts` runs after the runner has already died.

## Affected runs

| Pipeline | Job | Step duration | Resources tab | Final visible log line |
|---|---|---|---|---|
| [125962](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/125962) | [5094627](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/125962/workflows/2e93ac88-43fc-4ce2-b0d4-67177305bec5/jobs/5094627) | 194s | RAM pinned 100% for ~60s (OOM) | `PASS op-deployer/pkg/deployer/state (coverage: 63.8%)` |
| [125994](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/125994) | [5096684](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/125994/workflows/71187ad2-275f-46a8-bd83-fe12d3ef9077/jobs/5096684) | 46s | low memory throughout | identical |
| [125996](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/125996) | [5096833](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/125996/workflows/095f309c-6f0c-43ce-879f-2e10c7764538/jobs/5096833) | 84s | (presumed similar) | identical |

In every case, only parallel run index 4 fails; the other 11 shards succeed.

## Why we can't narrow further from CI data today

1. **Shard assignment is deterministic.** `justfile:283` splits `ALL_TEST_PACKAGES` by `awk 'NR % NODE_TOTAL == NODE_INDEX'` over space-separated *patterns*. For `NODE_INDEX=4, NODE_TOTAL=12` that's patterns at positions 4, 16, 28, 40 of the list, i.e.:
```
./op-core/...
./op-e2e/system/...
./op-e2e/actions/sync
./op-deployer/pkg/deployer/state/...
```
Confirmed by `Node 4/12 running packages: …` in every log. So shard 4 always gets this same bundle until the package list changes.

2. **Streamed log truncation is a gotestsum buffering artifact, not the kill point.** `--format=testname` only emits a line when a *package* completes. Across three runs of 46s / 84s / 194s, the same fast packages complete and none of the heavy `op-e2e/system/*` packages ever finish a result line, so the visible log ends at the same boundary regardless of when the kill actually happened.

3. **The jsonfile that would have per-test events isn't retrievable.** For both runs 5094627 and 5096684 (and presumably 5096833), `GET /api/v2/.../artifacts` returns indices `[0,1,2,3,5,6,7,8,9,10,11]` — no index 4. `store_artifacts: when: always` only protects against earlier-step failure, not against runner death. When the runner agent itself dies, every subsequent step is skipped, so `tmp/testlogs/log-4.json` and `tmp/test-results/results-4.xml` never leave the box.

## What we have narrowed it to

Of shard 4's 24 expanded packages (`go list ./op-core/... ./op-e2e/system/... ./op-e2e/actions/sync ./op-deployer/pkg/deployer/state/...`):

### Finished before kill in all three runs (8 PASS + 3 EMPTY)
- `op-core/devfeatures`, `op-core/eip1559`, `op-core/forks`, `op-core/interop/depset`, `op-core/interop/messages`, `op-core/predeploys`, `op-core/superchain`
- `op-deployer/pkg/deployer/state`
- EMPTY: `op-core/nuts`, `op-e2e/system/helpers`, `op-e2e/system/e2esys`

### Never emit a result line in any of the three runs (13 — the prime suspects)
```
op-e2e/system/altda op-e2e/system/fjord op-e2e/system/runcfg
op-e2e/system/bridge op-e2e/system/isthmus op-e2e/system/verifier
op-e2e/system/conductor op-e2e/system/p2p op-e2e/actions/sync
op-e2e/system/contracts op-e2e/system/proofs
op-e2e/system/da op-e2e/system/fees
```

The culprit is one or more packages from this list. With `go test -p` defaulting to `GOMAXPROCS=16` on `2xlarge.gen2`, up to 16 of these binaries can run concurrently, each spawning multi-node setups (geth/op-node/batcher/proposer/etc.) — plausible memory hot-spot for the OOM case. The non-OOM cases (46s / 84s) likely have a different mechanism (test panic killing process group, agent disconnect, subprocess deadlock with the parent test process timing out CI-side but not gotestsum-side, etc.) but converge on the same package neighborhood.

## Next step (already in flight)

#20965 switches gotestsum to `--format=standard-verbose` so per-test `=== RUN` events stream to the CircleCI log as they happen. The streaming log itself will then show which test was running at kill time — no artifact needed. Once that lands, the next occurrence of this should pin the failing test directly.

If after #20965 lands the failure is still ambiguous (e.g. the kill point lands between two RUN events), the next escalation is to drop `-p` to a low number for shard 4 only so package execution becomes serial, making "which package was running" trivially visible.

## Notes / context

- Job runs on `2xlarge.gen2` docker (16 vCPU, 32 GB) with `parallelism: 12`.
- Tests are *not* run with `-race`.
- All three runs share the identical truncation point — this is a buffering artifact, not the failure point.
- `--rerun-fails=3 --rerun-fails-max-failures=50` is configured but never fires here because the parent gotestsum process dies before it can orchestrate a rerun.

> **Claude:** This issue was authored by Claude with Adrian's input — investigation steps, links, and the package-list analysis were pulled together from CircleCI's v1.1 and v2 APIs across the three failed runs above.

Start with justfile:283 to understand shard 4's package assignment, then review issue #20965 and the affected CircleCI runs. Use the verbose per-test output to identify the package or test active at failure; if that remains ambiguous, investigate the proposed lower -p setting for shard 4. Done means the recurring failure is narrowed to a specific test or package and an actionable cause.

Written by the indexing model from the issue text.