huggingface / huggingface/transformers-ci
Triage: a retried dispatch reuses the start-of-job OIDC token and 401s ("Signature has expired"), dropping the group
- Dominant language
- Python
- Stars
- 1
- Forks
- 7
- Avg merge
- 1h 30m
- Merged PRs (30d)
- 39
Description
## Problem
A dispatch retry POSTs with the OIDC token minted at the start of the job, so a group that waits out
a provider-429 backoff is rejected and dropped.
Observed in run
[32126466463](https://github.com/huggingface/transformers/actions/runs/32126466463) (token minted
~10:23):
```
11:07:29 1b41b8147a294b6ea5914b6af709d5d8 ended with provider rate limit; retry 2/2 in 360.0s
11:13:50 [3/5] 3 integration tests for model `deepseek_vl` failing with `other` (other (3)) retry 2/2
11:13:50 fingerprint 35052fe37c21 → new PR
11:13:50 ✗ Serge POST /tasks failed: 401 Unauthorized
{"detail": "oidc_verification_failed: invalid OIDC token: Signature has expired", …}
11:13:50 dispatched 4/5 group(s) to Serge; 1 failed
```
## Cause
In `dispatch_serge_tasks` (`integration_failure_triage.py`), the refresh
```python
token = mint_serge_oidc_token() or token
```
sits at the top of the **polling** phase — after the inner dispatch `while pending and len(active) < limit`
loop, and behind `if not active: continue`. So:
- every `dispatch_to_serge(serge_url, token, …)` call uses a token minted at least one full cycle
earlier, and
- the 429-retry path (`time.sleep(sleep_for)` then `pending.append(...)`) re-POSTs after a 360s+ sleep
with that same token — here ~50 minutes old.
`reconcile_tracking_issue` already re-mints on every poll for exactly this reason (`:2519`: *"the one
minted at start can expire before every task finishes"*); the dispatch path never got the same
treatment. The longer the run (dispatch is throttled to 3 active tasks, so runs last 30-70 minutes),
the more likely any late POST is stale — not just retries.
Second-order effect: `401 oidc_verification_failed` is not rate-limit-shaped, so
`_SERGE_RATE_LIMIT_PAT.search(str(e))` fails, the group is abandoned (`failed += 1`) instead of
retried, and the tracking issue then shows a bare `⚠️ task failed` inherited from the *first*
attempt's 429 — which says nothing about a token having expired.
## Fix
1. Mint immediately before each `dispatch_to_serge` call (cheap: one local HTTP call to the Actions
token endpoint), or at minimum right after any backoff `sleep` and before re-queuing.
2. Treat `401` + `oidc_verification_failed` as retryable: re-mint once and re-POST rather than
dropping the group.
3. Surface it distinctly in the log/table (an expired-token drop is an infra problem, not a Serge
outcome), so it does not masquerade as `⚠️ task failed`.
Related: #79 (write outcomes from the dispatch loop) and #80 (a re-run wipes the recap) — together
those three are why a reader of the tracking issue cannot currently tell an infra drop from a real
task failure.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in integration_failure_triage.py at dispatch_serge_tasks, then compare its token handling with reconcile_tracking_issue, which already re-mints during polling. Trace dispatch_to_serge through the backoff and 401 paths. Done means late and retried dispatches can recover from expired OIDC tokens, and the logs or table distinguish an infrastructure token failure from a task failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100