huggingface / huggingface/serge

Let a task push its outcome to a GitHub issue (nightly triage table goes stale the moment the CI job exits)

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
51
Forks
10
Avg merge
11h 32m
Merged PRs (30d)
36

Description

## Problem

The nightly integration-failure triage (`huggingface/transformers-ci`) opens one tracking issue per
run — e.g. [transformers#48050](https://github.com/huggingface/transformers/issues/48050) — with a
row per dispatched group that is supposed to show the outcome. Serge cannot update it, so the table
is only ever as fresh as the dispatching GitHub Actions job:

- the job renders the table at creation (all `(pending)`), then polls `GET /tasks/…/status` in a
reconcile loop capped at `ITF_RECONCILE_TIMEOUT` (1200s) and exits;
- once the Action exits nothing writes to the issue again, so a task finishing after that window is
invisible until the next nightly run re-renders the table;
- Serge has no idea the issue exists. The triage payload passes `tracking_issue` only *inside the
prompt* ("include `Relates to #N` in the PR body"); there is no `tracking_issue` field anywhere in
`reviewbot/`. Serge's only outbound push on completion is Slack (`slack_notify_pr_created` /
`slack_notify_task_finished`), and `/status` is auth-only, so nothing outside a token-holding CI job
can even read the outcome.

Result: a `no_fix` or `error` that Serge knows about at 11:30 shows up on the issue tomorrow.

## Proposal

Add a GitHub-issue completion notification, symmetric to the Slack one:

```yaml
notifications:
github_issue:
repo: huggingface/transformers # defaults to the task's repo
number: 48050
marker: itf-9b5330bbfa6e # the caller's row key (group fingerprint)
```

On terminal state, hook it in next to `_notify_task_finished` (`reviewbot/webapp.py:1763`) and post
the outcome — status, one-line reason, PR link when there is one, `job_id`, and the normalizer error
when that is what stopped it. `GitHubClient.post_issue_comment`
(`reviewbot/github_client.py:269`) already exists and the endpoint is the same for issues, so a
comment path needs no new client code; if we want the *table row* itself updated in place, that needs
a small `PATCH /issues/{n}` body helper plus the `marker` above to find and rewrite exactly one row
idempotently (re-delivery must not duplicate).

Notes / constraints:

- Same failure-tolerance as the Slack path: best-effort, never fail the task on a notification error.
- The App token must carry `issues: write` for the target repo (the triage workflow already requests
it for itself).
- The target repo/issue must be validated against the task's own repo — do not let an arbitrary
caller make Serge write to any issue it has a token for.
- Keep it opt-in per task, off by default, so only the triage caller uses it.

## Why not solve it on the caller side

The CI job cannot outlive its runner, and the `fingerprint → job_id` map exists only in that job's
memory, so a periodic reconciler workflow would first need that state persisted (e.g. in a hidden
comment in the issue body). A push from Serge is the only path that covers tasks finishing after the
job exits. Tracked from the caller side in
huggingface/transformers-ci#79 and huggingface/transformers-ci#80, which fix the two staleness bugs
that are the caller's own fault.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in reviewbot/webapp.py around _notify_task_finished at line 1763 and inspect the existing Slack completion path, then read GitHubClient.post_issue_comment in reviewbot/github_client.py around line 269. Trace how task notification settings are represented and determine whether completion should add an idempotent issue comment or update a marked table row. Done means an opt-in terminal notification reports the outcome safely, validates the target repository, and cannot make notification failures fail the task.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.