mudler / mudler/vllm.cpp

Per-PR gates cannot see pairwise interactions: two independently-green branches can merge cleanly and still break main

Open
#2,848 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: ENG-CI

What

Every gate in this project proves that one branch is green against main. Nothing
proves that two independently-green branches are green together. Concurrently
open PRs can interact, and no gate in the tree can see it.

The concrete instance that surfaced it

Two branches, each green, each merging cleanly:

  • #2712 rewrites LtWorkspace in src/vt/rocm/rocm_matmul_hipblaslt.hip.
  • #2713 arms -Werror on the HIP compile language for the first time.

git merge-tree --write-tree returns rc=0 with no conflict — but that is a
textual result. After both land, #2712's new code is compiled under a flag it
has never been compiled under, and neither PR's own gate covers that. #2713
measured 19 warnings on the tree without #2712's rewrite; #2712 compiled its
rewrite without -Werror.

The implementer caught this unprompted and extended an already-queued job to build
the merged pair. That was judgement, not a gate — nothing in the tree would have
required it, and a less careful pair of branches would have landed a red main.

Why "clean merge" is not the property we need

merge-tree answers "do these diffs conflict textually". The property that matters
is "does the merged tree build and pass". Those differ whenever one branch changes
how code is compiled or checked and the other changes what code exists:

  • a branch arming a warning flag + a branch adding code
  • a branch tightening a checker + a branch adding a surface that checker reads
  • a branch narrowing a predicate + a branch adding a caller of it

The third shape already bit this repo today from the other direction: 6b97a6800
narrowed managed allocation and is not an ancestor of 9f3e6e223, the run whose
result the record then cited — so a landed commit invalidated a measurement's
premise without any gate noticing.

Not proposing a specific mechanism

Options span cheap-and-partial to expensive-and-complete: a merge-queue that builds
the combined tree; a CI job that builds main + each open PR pairwise (O(n²), likely
untenable at ~30 open PRs); a narrower rule that flags pairs touching an intersecting
file set OR where one changes build flags/checkers; or accepting the risk and relying
on main CI to catch it after the fact.

The last option is close to today's behaviour but is not stated anywhere, and the
main CI ratchet's base has been frozen since 2026-08-13, so "main CI catches it" is
weaker than it sounds.

Filing to name the gap, not to prescribe the fix. Whoever takes it should price the
options before choosing.

Not claimed

I have not measured how often concurrently-open PRs here actually intersect, nor
whether any red on main in the last month traces to this. Both are worth measuring
first, and either could show the risk is small enough to accept deliberately.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing CI gates and the main CI ratchet, then measure how often concurrently open PRs intersect and whether recent red main runs relate to invalidated assumptions. Compare the merge-queue, pairwise-build, targeted-pair, and deliberate-risk options; done means the risk and cost are measured and a mechanism or explicit acceptance decision is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.