[challenge]: Agentic verification of an experimental Rust tensor library: grow the canonical record of performance benchmarks and correctness oracles

Start by reviewing the tenferro-benchmark and tensor-ad-oracles repositories to understand their existing benchmark entries, workload specifications, and oracle provenance. Identify one uncovered workload or correctness gap, then measure it against the named references and record a reproducible result. Done means a merged benchmark entry or oracle case with full provenance, rather than an issue report alone.

Released by

Hiroshi Shinaoka, Saitama University

Contact email

shinaoka@mail.saitama-u.ac.jp

Method

Other

Challenge issue

Context. Agentic coding moves the bottleneck from writing code to trusting code. Trust requires continuously measuring two quantities against independent references: performance and numerical correctness. tenferro-rs is an experimental pure-Rust tensor library (einsum, linear algebra, eager and traced autodiff with complex-number support, CPU/GPU), largely agent-written. Its measurement infrastructure exists as two public repositories: tenferro-benchmark, a continuous performance benchmark against PyTorch, JAX, Julia, HPTT, and cuTENSOR, and tensor-ad-oracles, a machine-readable JSON database of derivative-correctness oracles generated from PyTorch.

Where the action is. Benchmark coverage is still far from sufficient. Realistic workloads, such as sequences of small and medium contractions, permutation-heavy einsum, eager-mode loops, and end-to-end autodiff training loops, are not yet covered systematically. Growing this coverage is this challenge.

The challenge. Use AI agents to grow the canonical record:

1. Add benchmark suites for the uncovered regimes: tensor-network contraction sequences (DMRG/TEBD-like sweeps), permutation-heavy einsum, small-tensor eager loops, and whole-program VJP training steps, each measured head-to-head against PyTorch and JAX.
2. For every gap found, fix it into the canonical record as a tracked benchmark entry with full reproduction (workload spec, sizes, hardware profile), so the gap stays visible until it is closed.
3. Diagnose: attribute gaps to their cause (GEMM backend, dispatch overhead, memory layout, missing fusion) with profiling evidence. A well-attributed gap is worth more than ten unexplained ones.
4. Correctness side: extend tensor-ad-oracles where coverage is thin (a documented JAX generator contract awaits implementation; higher-order derivatives, complex conjugation conventions, degenerate spectra) and run the oracles against tenferro-rs.
5. Bonus: prototype a fix for one attributed gap and show the benchmark entry improving.

What counts as a discovery. A performance gap or numerical discrepancy counts once it is fixed as part of the canonical record: a tracked benchmark entry, or a merged oracle case with full provenance. Issue reports alone do not count; merged cases do.

Scoring. Systematic contributions rank above single findings: a benchmark suite covering a workload regime ranks above a cluster of entries, which ranks above a single measurement. An attributed gap ranks above an unattributed one. A null result backed by new systematic coverage ("this regime is on par") is a valid contribution.

Recognition. Substantial contributions earn listing as a contributor to the tenferro-rs ecosystem. Substantial and sustained contributions will qualify for co-authorship on the planned tenferro-rs software paper, following standard authorship criteria.

Why this may lead to research output. Where and why a Rust tensor stack lags or matches the ML frameworks, measured systematically, is exactly the evidence base a foundational library needs, and the methodology of agent-driven performance metrology is a contribution to the AI agent infrastructure theme of this school. The collected suites remain public infrastructure that any tensor library can run against.