huggingface / huggingface/transformers-ci

Triage: publish terminal outcomes from the dispatch loop, not 15 minutes later in reconcile

Open
#79 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1
Forks
7
Avg merge
1h 30m
Merged PRs (30d)
39

Description

## Problem

The triage job learns each group's terminal outcome **inside the dispatch loop** (it is throttled to
3 active tasks, so it blocks and polls `/status` as it goes) and writes none of it to the tracking
issue. The only `update_issue_body` call is inside `reconcile_tracking_issue`
(`integration_failure_triage.py:2551`), which runs after the dispatch loop has drained.

Concretely, from run [32109614737](https://github.com/huggingface/transformers/actions/runs/32109614737)
(issue [transformers#48050](https://github.com/huggingface/transformers/issues/48050)):

```
07:06:23 [1/5] [2/5] [3/5] accepted
07:42:45 5435b310… terminal error: The repository's normalizer … rejected the patch
07:43:05 [4/5] accepted
07:47:48 67f2271a… terminal error: … normalizer …
07:48:08 [5/5] accepted
07:50:29 37b8d1bd… terminal error: … normalizer …
07:53:51 3f167da6… terminal status: no_fix
07:57:33 b63102e6… terminal status: no_fix
07:57:33 dispatched 5/5 group(s) to Serge
07:57:33 reconciling tracking issue #48050 for up to 1200s as Serge runs…
07:57:52 tracking issue #48050 refreshed: 5/5 resolved (0 PR, 5 no-fix/error) <-- the only PATCH
```

For 15 minutes the process knew three groups had failed while the issue still said `(pending)` for
all five. Anyone watching the issue during that window — the normal case, since dispatch dominates
the job's wall clock (51 of 55 minutes here) — sees a table that is knowably stale.

## Fix

Refresh the issue body from inside the dispatch loop whenever a group reaches a terminal status
(PR opened / `no_fix` / `error`), reusing the same render + PATCH path reconcile already uses, and
keeping the "only PATCH when the resolved set changed" guard so the number of writes stays bounded
by the number of groups.

The distilled outcome (`_distill_outcome`) is available at that point too, so the Outcome recap
lands with the row rather than 15 minutes later.

## Related

Once a re-run happens, those outcomes are lost again — see the sibling issue on
`_carry_forward_rows` dropping the Outcome recap.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in integration_failure_triage.py at reconcile_tracking_issue (line 2551), then trace the dispatch loop where terminal outcomes and _distill_outcome are available. Reuse the existing render and PATCH path, preserving the resolved-set guard; done means the tracking issue reflects each terminal group during dispatch and includes its Outcome recap.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.