Add gate on full-sweep / fail-fast runs to prevent wasted compute

Open
#2,103 0 comments 0 reactions 2 assignees View on GitHub

@adibarra is already working on this.

Since Jul 6, 2026.

Assessment

This issue has not been assessed yet.

Description

dev velocity enhancement p1

Summary

Add a gate on compute-heavy sweep runs (full-sweep-fail-fast, full-sweep, non-canary-full-sweep-enabled, etc.) to prevent obviously-wrong / wasteful runs from consuming compute before they start.

Motivation

Compute-intensive sweeps are expensive, and there is currently no guardrail to stop a run that is clearly misconfigured or violating known bad patterns. This wastes runner time, increases queue times, and blocks other engineers' work. Stopping wasted compute here is P1.

Related: this ties into the broader InferenceX compute-improvement effort of stopping people from sweeping trtllm/atom before submitting/validating vllm/sglang, which increases queue time and blocks vllm/sglang work (ATOM engineers wasting other AMD engineers' time).

Approaches discussed

  • FORBIDDEN_HACK.md allow/deny list (preferred starting point): Maintain a simple markdown file of forbidden patterns/hacks that we append to as needed. A gate reads it and denies a run when a known bad pattern is detected. Cheap, deterministic, low false-positive risk.
    • Run this gate only on fail-fast / canary runs (not every run) to limit overhead.
  • LLM-based gate (fallback / later): Call an LLM to evaluate whether a run should be denied.
    • Pros: catches novel bad patterns.
    • Cons: risk of false positives; cost. If pursued, use a cheaper model (e.g. Haiku / Sonnet 5 / fable5) rather than an expensive review model. Estimated ~$1–2 per fable call, cheaper than the ~$20/review Claude enterprise PR code review.

Notes

  • The CODEOWNER sign-off review is considered valuable and should stay.
  • Keep the gate scoped to fail-fast / canary runs initially to control cost.

Scope

Start with the FORBIDDEN_HACK.md + gate on fail-fast/canary runs. Evaluate an LLM-based check later if needed.

Dominant language
Python
Stars
1.7k
Forks
303
Avg merge
1d 13h
Merged PRs (30d)
284

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SemiAnalysisAI/InferenceX

All issues in SemiAnalysisAI/InferenceX

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.