compiler-explorer / compiler-explorer/infra

Trunk-compiler failure records persist forever in latest table (sticky 'failed before' across nightly rebuilds)

Open
#2,105 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
434
Forks
429
Avg merge
4d 21h
Merged PRs (30d)
38

Description

Spun out from #1342 since the perf work there is done and this is a separate correctness/staleness concern.

## Problem

When a trunk compiler (e.g. `clang_barry`, `gcc-trunk`, `clang_pawel`) fails to build a stable library release (e.g. `boost 1.85.0`, `fmt 10.0.0`), the failure record in the proxy's `latest` table persists indefinitely across nightly compiler rebuilds. The library-builder treats it as "still failing" and skips that combination forever — even after the underlying compiler bug is fixed.

## Why the existing freshness guard doesn't help

`has_failed_before` in `library_builder.py` and `fortran_library_builder.py` filters the result by comparing the stored `commithash` against `self.get_commit_hash()`:

```python
# library_builder.py
def has_failed_before(self):
return match_failed_build(self._get_failed_builds(), self.current_buildparameters_obj, self.get_commit_hash())
```

`match_failed_build` requires `commithash == current_commit_hash` for the failure to count. The intent is "stale records from a different commit don't poison new builds."

The flaw: `get_commit_hash()` returns the **library's** commit hash, not the compiler's. For:

- **Stable library releases** (boost 1.85.0, fmt 10.0.0, ...): the source tag commit is fixed. `get_commit_hash()` returns the same value forever.
- **Trunk compilers** (clang_barry, gcc-trunk): `compiler_version` is a stable string (`clang_barry`) that doesn't change when the binary is rebuilt nightly.

Result: `(library, library_version, compiler, compiler_version, arch, libcxx, compiler_flags)` is constant across compiler rebuilds, the `commithash` is constant for stable libraries, so `commithash == current_commit_hash` always passes and the failure record sticks.

Go and Rust have it worse — `has_failed_before` there passes `None` for the commit filter (the original `/hasfailedbefore` endpoint had no commit-hash check), so any failure is treated as current.

## Concrete incident

On 2026-05-04 during the perf-work session: `clang_barry`'s failure to build `boost_bin/1.85.0` from 2026-05-03 was still poisoning the 2026-05-04 run despite barry rebuilding nightly in between. Surfaced via #2090's `clear-for-compiler` row-count printout — a typoed compiler ID was silently no-opping until that PR landed.

## Code references

- Proxy `latest` schema: `migrations/001-initial.sql` in compiler-explorer/conanproxy. Composite PK `(library, compiler, library_version, compiler_version, arch, libcxx, compiler_flags)`. `commithash` added in `002-commithash.sql`.
- Failure write path: `BuildLogging.setBuildFailed` (`build-logging.js:93`).
- Failure clear path: `BuildLogging.setBuildFixed` (line 30) — only fires on a *successful* build of the same combo. If the combo never succeeds because we never retry, this never fires.
- Client-side filter: `match_failed_build` in `bin/lib/library_builder.py`, called from each builder's `has_failed_before`.

## Possible fixes (cheap → invasive)

1. **Server-side TTL on read.** Add `build_dt < N days ago` filter to the `getFailedBuildsForLibrary` query in `build-logging.js`. Failures older than the cutoff don't appear in `/failedbuilds` responses; the next builder run treats them as a clean retry.

- Pros: smallest change, no schema migration, no client work.
- Cons: doesn't reclaim disk; the rows are still there. Cutoff is server-policy, hard for client to override.

2. **Client-side TTL.** Have `/failedbuilds` include `build_dt` in the response, and `match_failed_build` ignore failures older than N days.

- Pros: lets policy live on the client; response shape is a small additive change.
- Cons: still doesn't reclaim disk. Two callers (Go/Rust currently bypass commithash filter entirely; same shape would need the TTL).

3. **Periodic server-side cleanup.** Cron job (or proxy-startup task) DELETEs `latest` rows where `success=0 AND build_dt < cutoff`. Plus a periodic `VACUUM`. Reclaims disk over time.

- Pros: simple, addresses the symptom and the disk bloat.
- Cons: doesn't help with rapid recovery — a failure recorded yesterday still blocks today's retry until cutoff.

4. **Include compiler binary identity in the failure key.** The real bug is that `compiler_version` (`clang_barry`) doesn't actually identify a unique compiler binary. Add the binary hash or build date to the key. A failure recorded for one binary doesn't apply to a freshly-rebuilt one.

- Pros: addresses the root cause.
- Cons: schema change in the `latest` table; every existing row's identity shifts; client + server need coordinated update; way bigger blast radius.

## Recommendation

Start with **(1)** — add `build_dt < cutoff` to the SELECT in `getFailedBuildsForLibrary`. With the covering index from conanproxy#75, the filter is effectively free. A 7-day cutoff would let nightly trunk compiler rebuilds retry stale failures without flooding the table with retry attempts mid-day.

If we want to also reclaim the disk over time, layer **(3)** on top as a separate maintenance migration / cron job. (1) handles the staleness for the read path; (3) handles long-term disk hygiene.

**(4)** is the right answer for correctness but is genuinely a bigger piece of work and probably not worth it unless the staleness symptom keeps biting after (1)+(3).

## Out of scope here

- The covering index, log-truncation, and bulk-fetch performance work on `/failedbuilds` are already done (conanproxy#74, #75, #76; infra#2100, #2104).
- The four-builder duplication (#1832 has a draft) is orthogonal.

Contributor guide

Open the contributing guide

Research direction

Start with getFailedBuildsForLibrary in build-logging.js and the latest-table schema in migrations/001-initial.sql and 002-commithash.sql. Trace how its response is consumed by match_failed_build in bin/lib/library_builder.py, including the Go and Rust paths. Done means failures older than the chosen cutoff no longer block retrying a compiler/library combination, while current failures still do.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python, sql
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.