huggingface / huggingface/serge
ITF task loop: inject prior discussion of the failure as context (`ghlore search --kind failure`), flagged off, non-gating
- Dominant language
- Python
- Stars
- 51
- Forks
- 10
- Avg merge
- 11h 32m
- Merged PRs (30d)
- 36
Description
## What
When a task session starts on an integration failure, inject **prior discussion of that failure** into the untrusted context: the error signature and test id, looked up against the indexed GitHub history with `ghlore`.
This is use case **#3 (root cause already documented)** from [`ghlore/docs/use-cases.md`](https://github.com/huggingface/transformers-ci-playbooks/blob/main/ghlore/docs/use-cases.md) §8a — trigger *"task loop, before patching"*, confidence **high**, ≥1 fully documented instance (#47938, the emu3 chain worked end to end in §4).
Related: #93 (the capability ask — let serge agents read PRs and issues; this is one implementation of it, scoped to the ITF case). Dispatch-side counterpart filed as huggingface/transformers-ci#118.
## Why this one first
§8a ranks four fix-side cases. This is the one with a documented instance where the answer existed and the agent could not reach it. §7d's turn-by-turn traces show the agent *searching* for exactly this and finding nothing — the history tool does not exist, so the turns are spent and the answer is not found.
The retrieval quality is measured, not assumed: on the `failure` slice of `ghlore`'s benchmark — error text and test ids, which is precisely this query — it scores **1.000** against a structural **0.000** for `grep`. `grep` cannot score here by construction: the answers are in issues and reviews, not in the tree.
## Where it goes
`reviewbot/tasks.py`. The precedent is already there — `format_pr_files_diff` (~line 445), the "prior attempt" context on an `existing_pr` follow-up, whose docstring is the right contract for this too:
> Best-effort and purely informational — not meant to be re-applied.
Same shape: a bounded blob, assembled before the loop, passed as `context` through `prompt_prefix_summary` so its size shows up in the existing per-turn accounting rather than becoming invisible prompt growth.
## The query
```
ghlore search "" --kind failure --test "" --repo huggingface/transformers
```
`--kind failure` implies the trust floor, so what comes back is what someone who could settle it said. Every hit carries author standing and age, which the agent already knows how to read from the review side.
## Guard rails
- **Flagged, default off.** serge needs a deploy to ship, so this can land dark and be enabled per-environment.
- **Untrusted context, fenced.** Retrieved text is data, never instructions — `ghlore` already wraps responses in its own untrusted envelope; keep it inside serge's untrusted block rather than unwrapping it.
- **Bounded** the way `format_pr_files_diff` is: a char limit and an explicit truncation marker. §7d's finding is that turns are the scarce resource — this must not trade a retrieval win for prompt bloat. Top ~3 hits, not the result set.
- **Fail soft.** Daemon down, empty result, timeout ⇒ no context block, loop unchanged. An empty result is the *common* case and must read as "nothing known", never as an error.
- **No gating.** This adds context. It must never decide whether to patch.
## Measurement
§10.3's **citation rate** is the pre-registered metric and it is the right one here, because it separates the two things that get conflated: *the tool was called* vs. *the tool helped*.
1. **Citation rate** — fraction of serge PR bodies that cite a retrieved thread a reviewer agrees was relevant. Requires the agent to be told it may cite; worth a line in the prompt.
2. **Turns to first patch**, against the current baseline. §7d's whole finding is that turns are where the budget goes; if context that answers the question does not reduce turns, that is a finding worth having.
3. **Expectation-hacking incidents must not rise** (§10.4). Prior discussion of a failure includes prior *rejected* fixes, which should push against expectation-hacking — but it needs watching rather than assuming.
Do **not** measure merge rate on this alone. §10.1's warning applies: several local fixes shipped recently (`e8ff196`, `b78b0d4`), and attributing a merge-rate move to retrieval without isolating them would be wrong in either direction.
## Prerequisite, and it is on the `ghlore` side
Two filters this depends on — `--error` and `--test` — are **the least exercised surface in the tool**. They are covered by the benchmark, but the last two cold agent evaluations ([ghlore#8](https://github.com/huggingface/ghlore/issues/8)) both record them as untested in real use. Worth running ten real ITF tracebacks through them before wiring anything, and filing what breaks on `ghlore`.
Also on that side: huggingface/ghlore#39 — the shipped agent skill currently tells agents *not* to use the code verbs, which is stale and would misdirect any agent that loads it. Cheap to fix, but fix it before pointing an automated consumer at the skill.
## Not in scope
Dispatch-time deferral (use case #2) is in the triage tool, not here — huggingface/transformers-ci#118. Review-side rationale retrieval (#10) and placement/idiom precedent (#13) are the two strongest §8a rows overall and belong in the review loop; they are deliberately not bundled here so this one stays measurable.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in reviewbot/tasks.py, reading format_pr_files_diff and the existing existing_pr follow-up context, then trace how prompt_prefix_summary carries context and how per-turn accounting is exposed. Run the mentioned real ITF tracebacks through ghlore's --error/--test filters before wiring the task-loop path. Done means a flagged-off, fail-soft, bounded informational context path with no gating and citation/turn measurements available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- ai, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100