RFC(eval): Distribute benchmark task groups across multiple VMs

## Problem

Maka Eval currently schedules task groups inside one runner process. Increasing `maxConcurrentTaskGroups` increases load on that execution host; it does not distribute a benchmark across a fleet of VMs. We want to shorten benchmark wall time by dynamically assigning task groups to multiple VMs while preserving existing evaluation and resume semantics.

This RFC proposes the first-stage boundaries and correctness contract. The scheduling backend remains open for discussion; this is not yet an implementation plan.

## Current behavior

At [a49ba7544](https://github.com/apache/maka/tree/a49ba7544c3bdd1ef648ec90c69b8f39f3e881c0):

- A cell is `task × repetition × subject`; a task group contains all subjects for one `task × repetition`.
- [`runExperiment`](https://github.com/apache/maka/blob/a49ba7544c3bdd1ef648ec90c69b8f39f3e881c0/packages/eval/src/runner.ts#L138) combines validation/preparation, local scheduling, result reuse, execution, and persistence. Its asynchronous workers share an in-memory queue index. A group holds its slot until all its cells settle.
- [`ExperimentExecutor`](https://github.com/apache/maka/blob/a49ba7544c3bdd1ef648ec90c69b8f39f3e881c0/packages/eval/src/runner.ts#L121) manages one local attempt through an operation callback. Harbor/Pier start local Python trial processes and use a localhost relay.
- [`FileAttemptStore`](https://github.com/apache/maka/blob/a49ba7544c3bdd1ef648ec90c69b8f39f3e881c0/packages/eval/src/attempt-store.ts#L26) holds an experiment-wide writer lock. Results are append-only; selection reuses the earliest valid, reusable attempt. Multiple remote runners cannot simply share this directory as a distributed work queue.

## Desired outcome and first-stage scope

A single benchmark run uses several pre-provisioned, homogeneous VMs. Each VM has a fixed number of task-group slots and claims another group when a slot becomes free. Work is assigned dynamically rather than statically partitioned by task count, since task durations vary.

The proposed first-stage semantics are:

1. **Keep task groups as the scheduling and capacity unit.** A group's pending subjects execute on one VM; its slot is released when the group settles. Optimizing within-group stragglers, scheduling individual cells, and fine-grained resource packing are out of scope.
2. **Persist results per cell.** Submit each completed cell without waiting for sibling subjects. On redispatch, skip cells with durably committed, valid reusable results and run only missing or infrastructure-failed cells. A committed zero score or subject failure is reusable too.
3. **Retry infrastructure failures, not poor benchmark outcomes.** Wrong answers, verifier zeroes, and benchmark subject timeouts with valid verification evidence are measured outcomes. VM loss or container startup failures are retry candidates. Missing verification or uncertain cleanup retains the existing infrastructure/indeterminate distinction. Retries are bounded; exhaustion leaves the evaluation incomplete, not a synthetic agent zero.
4. **Require matching execution environments.** Use the same VM class, execution revision, benchmark/toolchain versions, and image identities. Workers verify required environment identities before accepting work. Machine-local paths and credentials are resolved on the worker.
5. **Accept occasional duplicate execution after failures.** A partitioned VM may continue working while its replacement runs. Prevent duplicate result application; strict exactly-once model execution and VM fencing are not first-stage requirements.

## Proposed responsibility boundaries

| Concept | Responsibility |
| --- | --- |
| Experiment spec | Frozen benchmark, subjects, tasks, repetitions, budgets, and verification semantics |
| Run | One execution of a frozen spec, with execution identity and recorded scheduling/retry policy |
| Coordinator | Durable group state, capacity admission, assignment ownership, retries, and authoritative result selection |
| Worker | Advertise readiness/capacity, execute assigned groups through existing local adapters, and report results |
| Assignment | Group ownership for one dispatch, including an ID, generation, and expiry/liveness information |
| Attempt | Globally identified execution of one cell, with dispatch order, worker identity, result, and artifacts |

The coordinator is a logical responsibility: a Kubernetes-backed design may delegate part of it to existing controllers. VM provisioning and autoscaling remain external in the first stage.

Extract local cell/group execution from `runExperiment` and retain `SubjectAdapter` and Harbor/Pier lifecycle handling on the worker. Remote messages carry serializable assignment descriptions and results, not `ExperimentExecutor` callbacks. Local execution should reuse the same evaluation semantics.

## Failure, persistence, and recovery contract

- Persist ownership and attempt identities before dispatch. Claims must be atomic so healthy workers cannot independently acquire the same group generation.
- A missing heartbeat stops new assignments to that worker. Allow a recovery window before expiring ownership and redispatching unfinished work.
- Each dispatch has a new generation. Repeated uploads for the same attempt are idempotent; conflicting payloads are rejected. Results from superseded assignments must not overwrite authoritative results. Keep late evidence separately for diagnostics and attributable usage.
- Previously committed valid cell results survive group reassignment. Replacement workers obtain the current pending-cell set from authoritative state.
- A coordinator restart recovers assignments and committed results from durable state. A worker that finished but lost the acknowledgement can resend its submission safely.
- Store artifacts under attempt-specific identities. An accepted result references durable artifacts or explicitly records unavailable artifacts; worker-local absolute paths are not sufficient for fleet results.
- Report selected benchmark results separately from total observed execution usage. Duplicate or abandoned work may cost money; usage lost with a VM must remain unknown rather than be reported as zero.

Group placement is not an atomic statistical pair: partial retries can execute later, potentially on another VM. Preserve dispatch/worker provenance so analysis can distinguish these samples. Strict synchronized starts and automatic whole-group resampling are outside this proposal.

## Alternatives and open decisions

### Kubernetes Jobs or a lightweight coordinator/worker service?

- **Kubernetes:** evaluate a task-group Job whose worker invokes the existing local execution path. Kubernetes can provide placement and workload recovery, but we must establish compatibility with Harbor's Docker/Compose lifecycle, mounts, and networking. Application-level result identity and deduplication are still needed. [Kubernetes explicitly permits duplicate Job execution](https://kubernetes.io/docs/concepts/workloads/controllers/job/#handling-pod-and-container-failures).
- **Pre-provisioned VMs with a lightweight worker:** directly reuse the current host execution environment, adding durable claims, heartbeats, and result submission. This requires owning the small scheduling control loop.
- **Static sharding:** useful as a manual workaround, but leaves fast VMs idle behind slow shards and requires external recovery/result aggregation.

Choose the backend after checking existing deployment infrastructure and Harbor compatibility; do not implement two competing retry owners. If Kubernetes owns workload replacement, the application must reconcile those replacement executions with its assignment/attempt records.

### Policy and storage details

- Where should durable scheduling state and artifacts live?
- What heartbeat grace, retry backoff, and retry limits should be recorded for a run?
- How should the existing frozen `maxConcurrentTaskGroups` field map to the global run cap, alongside per-worker group slots? Preserve existing single-host behavior and record effective fleet concurrency.
- Is the shared provider quota sufficient for the desired fleet size? More VMs will not improve throughput once inference admission is saturated; elaborate provider quota scheduling is not required for this first stage.

## Acceptance scenarios

- Multiple matching workers dynamically claim groups, obey global/per-worker limits, and keep each group's pending subjects on one VM.
- A fast VM claims more work while another VM executes a slow group; within-group stragglers retain their group slot.
- VM loss after some cell commits preserves those results and retries only pending/retryable cells.
- An expired worker returns late, or a result acknowledgement is lost: authoritative results are not applied twice or overwritten.
- Coordinator restart preserves committed progress and recovers outstanding assignments.
- Valid zero scores and subject timeouts are retained; exhausted infrastructure retries appear as incomplete cells.
- Mismatched workers are rejected before evaluation work begins; accepted results and their available artifacts remain readable after VM removal.

Feedback requested on the responsibility boundaries, scheduling backend, and remaining policy/storage choices before turning this into an implementation plan.

Start by reading packages/eval/src/runner.ts and packages/eval/src/attempt-store.ts, then trace the existing SubjectAdapter and Harbor/Pier lifecycle paths. Compare the current local execution and persistence semantics with the RFC's acceptance scenarios, especially claims, retries, deduplication, and recovery. Done requires an agreed scheduling backend and implementation plan; the RFC explicitly leaves those decisions open.

Written by the indexing model from the issue text.