ethereum-optimism / ethereum-optimism/optimism

ci,rust: rebalance rust-cargo-hack partitioning (seeded shuffle is a no-op)

Open
#22,103 0 comments 0 reactions 1 assignee Claimed by @einar-oplabs View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

### Problem

`rust-cargo-hack` is the wall-clock driver of `rust-ci`, and its runtime is dominated by node imbalance rather than total work. Per-node timings from a recent run ([job 5384859](https://app.circleci.com/pipelines/gh/ethereum-optimism/optimism/jobs/5384859), parallelism 6):

| Step | Fastest node | Slowest node | Spread |
|---|---|---|---|
| `Run cargo hack` (lib, `--each-feature`) | 5.0 min | **26.7 min** | 5.3× |
| `Run cargo hack (test targets)` | 0.3 min | 5.9 min | 20.7× |

The job finishes when the slowest node does (~27 min); a balanced split of the same work would land near the mean (~15 min).

### Root cause: the seeded shuffle is a no-op

The `hack*` recipes in `rust/justfile` shuffle the package list with a seeded `shuf` and pass it as `-p` flags in shuffled order, intending to spread heavy crates across nodes. But cargo-hack discards `-p` ordering: `determine_package_list` (v0.6.44, [`src/main.rs:329`](https://github.com/taiki-e/cargo-hack/blob/v0.6.44/src/main.rs#L329)) iterates `workspace_members()` in cargo-metadata order and uses the `-p` values only as a **membership filter**. Since we pass every workspace package, the filter selects everything and the run order is byte-identical with `shuffle=true` and `shuffle=false`.

`--partition` ([`src/main.rs:171`](https://github.com/taiki-e/cargo-hack/blob/v0.6.44/src/main.rs#L171)) then slices that fixed sequence of (package × feature) runs into contiguous chunks weighted by feature count. Heavy crates sit adjacent in workspace-member order (the op-reth and kona blocks), so contiguous chunking concentrates them on the same nodes — hence the 5→27 min tail.

### Proposed fix

1. **Partition in the shell, not in cargo-hack.** The membership-filter behavior makes this easy: compute per-node package subsets in the recipe and pass each node only its own `-p` subset, dropping `--partition`. Assignment becomes fully ours to control.
2. **Weight the assignment.** Greedy longest-processing-time bin-packing over a small checked-in weight table (op-reth binaries and kona-host heavy; leaf protocol crates ≈ 1), ideally scaled by per-package feature count. Even a crude 3-tier table should collapse the spread to near the mean.
3. **Remove the placebo `shuffle`/`seed` parameters** from the `hack*` recipes and their CI call sites.

Bumping `parallelism` alone is a weaker alternative: it shortens the tail sublinearly (each node still compiles its chunk's dependency graph) and spends credits without fixing the skew.

Note: the two hack steps run serially in the same job, so packing should target the **sum** of both steps' per-package costs, not the lib step alone.

### Context

Observed while reviewing the test-targets gate added in #21501 (the imbalance predates that PR; the new step adds ~1–6 min/node and inherits the same partitioning).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.