mozilla / mozilla/blender

Close the loop on the fix flow: verify fixes against runnable checks before committing

Open
#68 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6
Forks
2
Avg merge
1d 11h
Merged PRs (30d)
17

Description

Summary

BLEnder's fix flow is one-shot. Claude reads CI logs, writes a fix, writes a commit message, and dies. The real CI runs after the commit. Claude never sees whether its fix turned the build green. If the fix is wrong, nothing inside the run catches it.

This issue proposes turning the fix flow into a verify loop: Claude fixes, runs the checks it can run, and iterates until they pass or the budget runs out. This applies the "compounding correctness" idea — spend tokens until a verifiable signal says the work is correct, instead of spending a fixed number of turns and hoping.

Problem

  • prompts/fix-ci-prompt.md:28 tells Claude "You have a limited number of turns. Be direct." That discourages iteration.
  • config/defaults.yml caps the fix at max_claude_turns: 30 and max_budget_usd: 2.00. These cap by turns, not by verifiable progress.
  • The fix is committed (.github/workflows/fix-dependabot-pr.yml, "Commit and push fix" step) with no in-run confirmation that the failing check now passes.

A wrong fix gets committed. The target repo's CI then re-fails. At best the 30-minute sweep re-picks the PR; at worst a human debugs it. Either way the pipeline spent tokens without locking in correctness.

Proposed change

Make the fix flow loop on a runnable verification signal:

  1. Reproduce the failing check.
  2. Make a fix.
  3. Re-run the check.
  4. Green → write commit message and exit. Not green → read the new error and fix again.
  5. Repeat until green or the budget is exhausted.

Constraints to respect:

  • The fix sandbox has no network and no credentials (see README "Security model"). So only checks that need no network can run in-loop: type checks, unit tests, linters. That covers most Dependabot breakage. Checks that need network (e.g. integration suites) stay out of scope and the loop should not block on them.
  • Keep the existing fix validation (sensitive-file rejection, secret scan) as the commit gate. The verify loop is additive.
  • Do not add a separate "review the fix" agent. The repo's own runnable checks are the quality signal — use them as the loop's exit condition.

Out of scope (note, don't implement)

  • Model tiering (route easy fixes to a cheaper model, hard ones to Opus).
  • Replacing the flat turn/budget caps with a distance-from-green budget.

These are follow-ups. This issue is just: close the loop.

Acceptance criteria

  • The fix flow re-runs the relevant runnable check(s) after a fix and iterates if they still fail.
  • A fix is committed only after the in-run checks pass (or the budget is spent, in which case the existing "could not fix" path applies).
  • prompts/fix-ci-prompt.md is updated from "be direct, limited turns" to "iterate until the check is green."
  • Behavior is covered by tests under tests/.

Context

This came out of a discussion mapping "tokenmaxxing," "loops," and "compounding correctness" onto BLEnder. BLEnder is a token pipeline; the value only compounds if each iteration starts from verified-good state. Closing the loop on the fix flow is the single highest-leverage change toward that.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading .github/workflows/fix-dependabot-pr.yml, prompts/fix-ci-prompt.md, and config/defaults.yml to trace the existing fix and commit flow. Then inspect tests/ for coverage of this behavior and run the relevant tests. Done means runnable checks are retried after fixes, commits wait for a passing signal or the existing budget-exhausted path, and the prompt and tests reflect the loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, devops, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.