huggingface / huggingface/transformers-ci

Record which caller workflow and which GitHub event produced a run (additive ci_github_event + ci_caller)

Open
#92 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
7
Avg merge
1h 30m
Merged PRs (30d)
39

Description

## Problem

CI telemetry records a run's `ci_event` family, but not **which workflow called it** nor **which GitHub event triggered it**. `ci_event` is a human-typed Slack-report input that we borrowed as a stream key, and it does not uniquely identify a caller.

Two concrete consequences today:

1. **`Daily CI` is passed by two different callers.** `self-scheduled-caller.yml` and `self-scheduled-flash-attn-caller.yml` both pass `ci_event: Daily CI` → both slugify to exactly `daily`, both run `run_models_gpu`, and both are scheduled at the same cron (`17 2 * * *`). The daily dashboard's `$latest_run_id` variable is

```promql
query_result(topk(1, last_over_time(pytest_run_start_time_seconds{ci_event="daily"}[90d])
or timestamp(last_over_time(pytest_run_active{ci_event="daily"}[5m]))))
```

so whichever of the two starts a second later becomes "the latest daily run" and takes over the Latest-Run-Jobs panels.

2. **No trigger event is stored at all.** `schedule` / `push` / `workflow_dispatch` / `repository_dispatch` are indistinguishable. The variable above also has no `pr` filter, so a debug push to a `run_nvidia_ci*` branch outranks the real scheduled run for as long as it is the most recent.

## Current state (baseline `a6d8563`)

- The only caller signal is set in YAML, in `.github/workflows/daily-ci_reusable_model_job.yml:234-239`, which slugifies the caller's `ci_event` input into `transformers.test.ci_event` (plus `transformers.test.hardware`).
- `src/transformersci/otel/cli.py` (`build_resource_attributes`) emits provider / job / run id / `cicd.pipeline.task.name` / `vcs.ref.head.name` / `vcs.ref.head.revision` / `vcs.change.id` / `vcs.change.url` / `vcs.repository.name`. **`GITHUB_EVENT_NAME`, `GITHUB_WORKFLOW` and `GITHUB_WORKFLOW_REF` are read nowhere in the repo.**
- The exporter reads `ci_event` in `extract_trace_rows` (`trace_exporter.py:1840`, default `"none"` at `:1922`) and puts it on the run/job rollup label sets (`:3017`, `:3073`); the PR badge splits streams on it (`RUN_SLOW_CI_EVENT` at `:192`, `_badge_event_matches` at `:4044`). That part works and this issue does not touch it.
- PR CI (`pr-ci_reusable_test_job.yml`) is instrumented but sets no `OTEL_RESOURCE_ATTRIBUTES`, so every PR-CI run reports `ci_event="none"`.
- Neither `ci_event` nor `hardware` reaches the published Parquet (`src/transformersci/publish/tables.py`).

## Proposal — two additive labels; `ci_event` is frozen

**Backward compatibility is the hard constraint.** `ci_event` keeps its exact current meaning, values and writer; `cli.py` must never emit it. Every query that works today keeps working, and the existing history stays queryable as-is.

Add two new attributes in `cli.py`, sourced from env vars Actions already injects — so **no caller workflow in `huggingface/transformers` needs to change**:

| resource attribute | Prometheus label | source | examples |
|---|---|---|---|
| `transformers.test.ci_github_event` | `ci_github_event` | `GITHUB_EVENT_NAME` | `schedule`, `push`, `workflow_dispatch`, `repository_dispatch`, `issue_comment`, `pull_request`, `merge_group` |
| `transformers.test.ci_caller` | `ci_caller` | workflow **file** basename from `GITHUB_WORKFLOW_REF`, falling back to `GITHUB_WORKFLOW` | `self-scheduled-caller`, `self-scheduled-flash-attn-caller`, `self-nightly-caller`, `self-comment-ci`, `pr-ci-caller`, `push-important-models` |

Rationale:

- In a reusable workflow, `GITHUB_WORKFLOW` / `GITHUB_WORKFLOW_REF` resolve to the **top-level caller** — exactly the missing dimension, and it separates flash-attn from daily for free. The event alone cannot: both are `schedule`.
- File basename, not display name: display names get renamed, the path is the stable identity.
- No token, no API call, no secret — unlike the GitHub-token paths, this cannot silently degrade.
- Purely additive label *names*, so no existing series changes meaning. `ci_event` stays the human-facing family; the new labels are the machine-facing truth. They are allowed to disagree — for flash-attn, `ci_event="daily"` with `ci_caller="self-scheduled-flash-attn-caller"`. That disagreement is the bug made visible.
- PR CI gains identity without gaining a `ci_event`, so the badge's `ci_event!="pr-comment"` stream logic is untouched.

## Phases

**P1 — producer (`cli.py`), no workflow edits.** A `slugify_label()` helper; append the two attributes on the `github_actions` branch; parse `GITHUB_WORKFLOW_REF` (`owner/repo/.github/workflows/.yml@refs/...` → split on `@`, basename, drop `.yml`, slugify), falling back to `GITHUB_WORKFLOW` then `unknown`. CircleCI/local emit `unknown` / `circleci` / `local` so the exporter never special-cases a provider. Tests: the ref parse (including `refs/pull/N/merge`), the fallback, both-unset, and an assertion that `cli.py` never emits `transformers.test.ci_event`.

**P2 — exporter: read and promote.** Mirror exactly how `ci_event` / `hardware` are already handled: `extract_trace_rows` (`:1780` init, `:1840` read, `:1922` return, default `unknown`), then the run/job aggregates (`:2904`, `:2946`) onto the run-scoped label sets (`:3017`, `:3073`). **Run-scoped only** — never on per-test series (the cardinality trap already paid for once with `run_id`). `pytest_run_info` is one-series-per-run, so the two labels cost zero extra series there. Add a golden test that `ci_event`'s emitted output is byte-identical.

**P3 — fix the collision in the dashboards, compatibly.** The trap: adding a matcher for a label that did not exist drops all history — `{ci_caller="self-scheduled-caller"}` matches nothing older than the deploy, so 90 days of daily runs would vanish. Use the empty-alternative form, the same trick already used for `ci_event` at `trace_exporter.py:4172-4178`:

```promql
{ci_event="daily", ci_caller=~"self-scheduled-caller|"}
```

Apply it to `$latest_run_id` (plus `ci_github_event=~"schedule|"` or `pr="main"`) and the other 16 `ci_event="daily"` selectors. Add `Trigger` / `Caller` columns to the run tables, following the existing `Event` column pattern in the by-PR dashboard. A `$caller` template variable is a deliberate follow-up, not part of this change.

**P4 — optional backfill of the existing 90 days.** `run_id` *is* the GitHub run id, and `GET /repos/{repo}/actions/runs/{run_id}` returns `.event` and `.path`. The exporter already calls that endpoint family via `fetch_github_run_activity_cached` (`:2119`) — reuse its cache. Fill the two labels **only when the span attributes are absent**, behind an env flag (default off), accepting `unknown` for runs aged out of the window. Once backfilled, the P3 matchers can be tightened to exact matches.

**P5 — publisher.** Append `ci_event`, `hardware`, `ci_github_event`, `ci_caller` to `RUN_ROLLUP_COLUMNS` (at the end, to keep the Parquet schema additive) plus a data-card note. `ci_event` and `hardware` already exist in the exporter and were simply never published.

**P6 — docs.** The comment block at `daily-ci_reusable_model_job.yml:95-102` is accurate about the mechanism but claims every caller family "lands on its own label value instead of all reporting as `daily`" — which flash-attn disproves. Correct it and point at the new attributes.

**P7 — deferred: extend instrumentation.** Out of scope here. Only two workflows call `configure-ci-otel` (`daily-ci_reusable_model_job.yml` and `pr-ci_reusable_test_job.yml`); inside `daily-ci_reusable.yml`, `run_pipelines_torch_gpu`, `run_examples_gpu`, `run_torch_cuda_extensions_gpu`, `run_quantization_torch_gpu` and `run_kernels_gpu` run plain `pytest` and emit nothing. Once `ci_caller` exists these gaps become *measurable* — a run whose caller we can see but whose jobs emit no spans is a visible hole rather than an absence.

## Verify before writing exporter code

Confirm `GITHUB_WORKFLOW_REF` resolves to the top-level caller under PR CI's double nesting (`transformers/pr-ci-caller.yml` → `pr-ci_dynamic_caller_example.yml` → `pr-ci_reusable_test_job.yml`) and not to an intermediate. One `echo "$GITHUB_WORKFLOW_REF / $GITHUB_WORKFLOW"` in the instrumented step on a debug push settles it.

## Acceptance criteria

1. Every new instrumented run carries `ci_github_event` and `ci_caller`, with no change to any caller workflow in `huggingface/transformers`.
2. `ci_event` values are byte-identical before and after; no query that works today returns different results, and no dashboard needs editing to keep working.
3. The daily dashboard's latest-run panels are immune to flash-attn runs and to debug pushes, without losing pre-deploy history.
4. "Which caller and which GitHub event produced this run?" is answerable from Prometheus alone, and from the published Parquet for the current window.
5. Per-test series cardinality is unchanged.

## Appendix — every `ci_event` value in `huggingface/transformers`

Reaching instrumented code (only `run_models_gpu` and `run_trainer_and_fsdp_gpu` are instrumented):

| caller | `ci_event` | slug | `ci_caller` (new) |
|---|---|---|---|
| `self-scheduled-caller.yml` | `Daily CI` | `daily` | `self-scheduled-caller` |
| `self-scheduled-flash-attn-caller.yml` | `Daily CI` | `daily` (collides) | `self-scheduled-flash-attn-caller` |
| `self-nightly-caller.yml` | `Nightly CI` | `nightly` | `self-nightly-caller` |
| `self-comment-ci.yml` | `PR Comment CI` | `pr-comment` | `self-comment-ci` |
| `self-past-caller.yml` | `Past CI - ${framework}-${version}` | `past-ci-tensorflow-2.5` … `-2.11` | `self-past-caller` |
| `push-important-models.yml` | `push` | `push` | `push-important-models` |
| `pr-ci-caller.yml` | *none set* | `none` (unchanged) | `pr-ci-caller` |

Declared but never reaching instrumented code (these callers target `huggingface/hf-workflows` or local uninstrumented reusables), so no attribute work applies to them: `self-scheduled-amd-mi250-caller.yml`, `-mi300-`, `-mi355-`, `self-scheduled-intel-gaudi3-caller.yml`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by verifying GITHUB_WORKFLOW_REF and GITHUB_WORKFLOW under PR CI's nested reusable workflows, as requested. Then read src/transformersci/otel/cli.py and the cited trace_exporter.py extraction and aggregation points, followed by the existing ci_event tests and dashboard selectors. Done means additive caller and event labels, preserved ci_event output and series cardinality, compatible dashboard queries, and published Parquet columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, prometheus, python
Domain
ci-cd, devops, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.