huggingface / huggingface/transformers-ci

Triage: a retried dispatch reuses the start-of-job OIDC token and 401s ("Signature has expired"), dropping the group

Open
#83 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1
Forks
7
Avg merge
1h 30m
Merged PRs (30d)
39

Description

## Problem

A dispatch retry POSTs with the OIDC token minted at the start of the job, so a group that waits out
a provider-429 backoff is rejected and dropped.

Observed in run
[32126466463](https://github.com/huggingface/transformers/actions/runs/32126466463) (token minted
~10:23):

```
11:07:29 1b41b8147a294b6ea5914b6af709d5d8 ended with provider rate limit; retry 2/2 in 360.0s
11:13:50 [3/5] 3 integration tests for model `deepseek_vl` failing with `other` (other (3)) retry 2/2
11:13:50 fingerprint 35052fe37c21 → new PR
11:13:50 ✗ Serge POST /tasks failed: 401 Unauthorized
{"detail": "oidc_verification_failed: invalid OIDC token: Signature has expired", …}
11:13:50 dispatched 4/5 group(s) to Serge; 1 failed
```

## Cause

In `dispatch_serge_tasks` (`integration_failure_triage.py`), the refresh

```python
token = mint_serge_oidc_token() or token
```

sits at the top of the **polling** phase — after the inner dispatch `while pending and len(active) < limit`
loop, and behind `if not active: continue`. So:

- every `dispatch_to_serge(serge_url, token, …)` call uses a token minted at least one full cycle
earlier, and
- the 429-retry path (`time.sleep(sleep_for)` then `pending.append(...)`) re-POSTs after a 360s+ sleep
with that same token — here ~50 minutes old.

`reconcile_tracking_issue` already re-mints on every poll for exactly this reason (`:2519`: *"the one
minted at start can expire before every task finishes"*); the dispatch path never got the same
treatment. The longer the run (dispatch is throttled to 3 active tasks, so runs last 30-70 minutes),
the more likely any late POST is stale — not just retries.

Second-order effect: `401 oidc_verification_failed` is not rate-limit-shaped, so
`_SERGE_RATE_LIMIT_PAT.search(str(e))` fails, the group is abandoned (`failed += 1`) instead of
retried, and the tracking issue then shows a bare `⚠️ task failed` inherited from the *first*
attempt's 429 — which says nothing about a token having expired.

## Fix

1. Mint immediately before each `dispatch_to_serge` call (cheap: one local HTTP call to the Actions
token endpoint), or at minimum right after any backoff `sleep` and before re-queuing.
2. Treat `401` + `oidc_verification_failed` as retryable: re-mint once and re-POST rather than
dropping the group.
3. Surface it distinctly in the log/table (an expired-token drop is an infra problem, not a Serge
outcome), so it does not masquerade as `⚠️ task failed`.

Related: #79 (write outcomes from the dispatch loop) and #80 (a re-run wipes the recap) — together
those three are why a reader of the tracking issue cannot currently tell an infra drop from a real
task failure.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in integration_failure_triage.py at dispatch_serge_tasks, then compare its token handling with reconcile_tracking_issue, which already re-mints during polling. Trace dispatch_to_serge through the backoff and 401 paths. Done means late and retried dispatches can recover from expired OIDC tokens, and the logs or table distinguish an infrastructure token failure from a task failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.