huggingface / huggingface/transformers-ci

Triage: a same-day re-run keeps the ⚠️/🚫 markers but wipes the Outcome recap that explains them

Open
#80 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1
Forks
7
Avg merge
1h 30m
Merged PRs (30d)
39

Description

## Problem

A same-day re-run keeps the `⚠️ task failed` / `🚫 no fix` markers but destroys the explanation they
promise.

`_render_outcome_recap` (`integration_failure_triage.py:2047`) builds the recap only from
`details[fp]` for **this run's** `targets`, and `_carry_forward_rows` (`:2240`) carries only the
*table row markdown* for prior groups. So the reason, the LLM + token spend, and the
`_render_normalizer_block` collapsible with the normalizer output tail all vanish on the next render.

Observed on [transformers#48050](https://github.com/huggingface/transformers/issues/48050):

- run at 07:03 → recap published 07:57:52 with three normalizer failures explained (which checker,
the output tail);
- manual re-run at 10:22 → body re-rendered 10:23:42; two rows still read `⚠️ task failed`, the
recap section is gone entirely, and there is now no way to find out why without the Actions log
or a Serge dashboard round trip.

Two smaller defects in the same function:

1. **Carry is keyed on model name**, not fingerprint (`model not in current`). Today's run has two
`generation` groups (`output_mismatch` and `import_or_config`), so the *prior* `generation` row is
treated as current and dropped — even though it was a different failure group.
2. A prior `(pending)` row is discarded unconditionally. That is right for a group being
re-dispatched, wrong for a group this run simply did not pick — it silently disappears from the
table instead of staying pending.

## Fix

- Carry the recap rows (and their normalizer collapsibles) forward the same way table rows are,
parsed out of the existing body; or persist the per-fingerprint outcome in a hidden HTML comment in
the body so a re-render can rebuild the recap exactly.
- Key both carries on the group **fingerprint** (already in the row's branch name /
`serge/fix/itf-`), not the model name.
- Keep a prior `(pending)` row when its fingerprint is not among this run's targets.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in integration_failure_triage.py at _render_outcome_recap (line 2047) and _carry_forward_rows (line 2240), then trace how the existing body is parsed during a re-render. Verify that recap explanations and normalizer collapsibles survive, carries are keyed by fingerprint rather than model name, and untouched pending rows remain present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.