fix pr-merge relitegation papercuts

Description

The nightly pr-merge relitigation repaints the check on commits nobody has touched, and it is mostly wrong when it does.

Three weeks of CI-Public/pr-merge status history (187 PR heads, 632 builds, 2026-08-21 -> 2026-09-10):

	builds	red
First pr-merge run on a commit	187	12%
Nightly reruns of those same commits	445	23%
master job, same pipeline, same window	100	2%

23 PRs were flipped green -> red on code nobody had pushed to. 22 of those were the CI hiccuping: the build died before any test ran, a test timed out, or in one case 4,241 targets went NO STATUS at once. 13 of them went back to green on their own, same commit, no push.

The exception is the interesting part. #11123 is a real catch: pr-head green, pr-merge red on //test:jpeg_sky130hd-tcl_test (Metrics do not satisfy limits: DRT::ANT::errors 1 > 0), every night since 09-02. That is the signal pr-merge exists for, and note what made it legible: nine identical failures in a row, not one red X.

Suggested Solution

One true positive in three weeks? Perhaps just drop pr-merge litegation and accept and fix the occasional master regression. Regressions happen today and it is dealt with. 1 more regression in three weeks is perhaps the best way. Quality is not about perfection, it is making sound choices.

Or perhaps... leave the first pr-merge run alone. That check is worth keeping and is where nearly all the true signal lands.

Have the nightly relitigation post its result as a PR comment instead of overwriting the status. Then #11123 shows up as "failed 9 nights running on jpeg_sky130hd", and the rest shows up as what it is, without a red check on a commit that was green yesterday and will be green tomorrow.

Additional Context

Method and full per-PR data

Method. For every open and recently closed PR I pulled the full CI-Public/pr-merge commit-status history for its head SHA from the GitHub API. Because each nightly relitigation is a distinct Jenkins build of the same, unchanged commit, the status history gives a sequence of verdicts per commit. For each red verdict I pulled the tail of the Jenkins console log and extracted the failing bazel targets. Window: 2026-08-21 → 2026-09-10, bounded by Jenkins log retention. 187 PR heads, 632 pr-merge builds.

Categories

	Category	What it means	Signature in the data
C1	True positive	The PR merged with current master genuinely breaks something. Actionable by the PR author.	A named test target fails, and keeps failing on every relitigation until the PR or master changes.
C2	False positive — CI instability	Agent/container lost, network, image pull, bazel dies, test timeout. Nothing to do with the PR.	Build ends red with no failing test, or thousands of targets go NO STATUS at once, or a test TIMEOUTs. Same commit is green before and after.
C3	False positive — real failure unrelated to the PR	master itself is broken, or a test is broken/flaky for everyone.	The same failure hits several unrelated PRs the same night.
C4	Unknown	Jenkins discarded the log before I looked.	—

How often relitigation changes a verdict

	builds	red	red rate
First pr-merge run on a commit	187	23	12%
Nightly relitigations of those same commits	445	103	23%
master branch job, same window	100	2	2%

The master job runs essentially the same pipeline (it adds clang-tidy), and both of its non-green builds were a Jenkins Timeout after 180 seconds and one flaky test — so the pipeline against known-good code is red 2% of the time, while relitigating unchanged PR commits is red 23% of the time.

Every PR that relitigation turned from green to red on unchanged code

. = green, X = red, one character per nightly build of the same commit — no pushes, no rebases in any of these rows.

PR	nightly pr-merge verdicts on the same, unchanged commit	cause of the red run(s)
#9097	..X.X............. (08-21→09-10)	C4 log discarded ×2
#10545	.......X (08-21→08-28)	C2 timeout: //src/cgt/test:ibex_sky130hd-tcl_test
#10847	...X..XX.......... (08-22→09-10)	C4 log discarded ×3
#10983	..X. (08-21→08-24)	C4 log discarded
#11025	..X..... (08-21→08-28)	C4 log discarded
#11061	...........X (08-21→09-03)	C4 log discarded
#11070	...........X.. (08-21→09-05)	C4 log discarded
#11121	.X.. (08-25→08-28)	C4 log discarded
#11123	.XXXXXXXXX (09-01→09-10)	C1 //test:jpeg_sky130hd-tcl_test FAILED ×7, C4 ×2
#11196	.....X....X. (08-22→09-04)	C4 log discarded ×2
#11211	......X..... (08-27→09-10)	C2 timeout: //src/grt/test:repair_antennas_post_drt_cugr-tcl_test
#11225	.......X (09-03→09-10)	C2 build died, no test failed
#11270	...X... (08-30→09-06)	C4 log discarded
#11283	....X.... (08-31→09-09)	C2 build died, no test failed
#11286	..X. (08-31→09-04)	C4 log discarded
#11303	..X.....X (09-02→09-10)	C2 build died, no test failed ×2
#11311	......X (09-03→09-10)	C2 build died, no test failed
#11317	.......X (09-02→09-10)	C2 build died, no test failed
#11320	.......X (09-02→09-10)	C2 build died, no test failed
#11323	...X..X (09-04→09-10)	C2 build died; C2 whole build collapsed, 4241 targets FAILED/NO STATUS
#11352	..X (09-07→09-10)	C2 build died, no test failed
#11355	..X (09-08→09-10)	C2 build died, no test failed
#11362	.X (09-09→09-10)	C2 build died, no test failed

Summary of those 37 red runs: C1 7, C2 14, C4 16 — and 13 of the 23 PRs went back to green on the same commit with no push, which settles them as false positives whatever the discarded log said. Most of the remaining ones flipped red for the first time on 09-10 and simply have not had another nightly run yet.

One true positive in three weeks: #11123 (gpl: accumulate nonPlaceArea before multiplying density). pr-head is green, pr-merge has failed every night since 09-02 on //test:jpeg_sky130hd-tcl_test with Metrics do not satisfy limits: DRT::ANT::errors 1 > 0. That is exactly the signal pr-merge exists for — and note how it announced itself: the same failure, reproducibly, every night, not one red build.

All 126 red pr-merge runs in the window, by cause

cause	red runs
A named test target failed	55
Jenkins log discarded before I could look (C4)	42
Build died with no test failure (C2)	15
Whole build collapsed, >20 targets FAILED/NO STATUS at once (C2)	10
Test timeout/abort only (C2)	4

The named-test failures are not spread across the PR population: 55 of the 63 are on PRs that were already red on their very first pr-merge run, where the author was told on day one and relitigation adds nothing. Only 8 are on PRs that pr-merge had passed, and 7 of those 8 are #11123.

The most frequently failing targets across all red runs — //:fmt_tcl_test (17), //:lint_bzl_test (17), //test:aes_sky130hd-tcl_test (13), //:fmt_bzl_test (12), //test:jpeg_sky130hd-tcl_test (11) — are formatting/lint checks and flow tests that recur across unrelated PRs, which is the C3 shape: the PR is not what changed.

The reds are correlated across unrelated PRs

Distinct PRs red per day, out of all pr-merge builds that day:

date	builds	distinct PRs red
08-21	31	7
08-25	25	4
09-03	29	8
09-04	42	9
09-05	33	9
09-08	43	11
09-09	57	11
09-10	63	23

18 of the 19 days with any red had two or more unrelated PRs red. 23 unrelated PRs did not each independently break on 09-10 — that day the master job failed too, on a Jenkins step timeout.

Why this matters for the proposal

A single red pr-merge carries almost no information: 4% of the PRs it newly reddened had a real problem. What does carry information is persistence — the same named test failing night after night, like #11123. A comment listing the recent nightly results makes persistence visible and legible to the reviewer, while a red check mark on a green commit does not, and costs every author an investigation into someone else's infrastructure.

Start by tracing the CI-Public/pr-merge nightly relitigation entry point and its Jenkins status handling, then review the GitHub API behavior used for commit statuses and PR comments. Done means the first pr-merge result remains intact, nightly results are posted as a PR comment, and repeated failures such as #11123 are visible without overwriting the commit status.