huggingface / huggingface/diffusers
PR GPU CI skips new-model GPU tests: Cosmos3 memory tests failed on an unrelated PR
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
## Summary
GPU-only model tests added in a PR can silently never run on that PR, then surface as failures on an **unrelated** PR later. This just happened with the Cosmos3 transformer: two `TestCosmos3OmniTransformerMemory` tests failed in the ace-step-lora PR's CI ([run](https://github.com/huggingface/diffusers/actions/runs/29635377742/job/88057065236)), even though ace-step-lora has nothing to do with Cosmos3.
This is relevant to the ongoing work on **selecting which tests to run for a given PR** — the fix here is really "make the PR GPU workflow run the tests that the PR's changes actually affect."
## What happened
| PR | Added | Ran its Cosmos3 GPU tests? |
|---|---|---|
| #13818 "Adding Cosmos 3" | `transformer_cosmos3.py` + AE + pipeline + docs + examples, but **zero `tests/` files** | Workflow triggered (via `examples/**`), but no transformer tests existed yet |
| #14181 "Cosmos3 edge support" | **First** added `tests/models/transformers/test_models_transformer_cosmos3.py` incl. `TestCosmos3OmniTransformerMemory` | **No** — GPU workflow never triggered (see below) |
| #14193 "ace-step lora" | Touched a loader file | GPU workflow triggered → ran the full model suite → Cosmos3 memory tests ran for the first time → **failed**, blocking an unrelated PR |
## Root cause — two compounding gaps
**1. `pr_tests_gpu.yml`'s `paths:` filter is out of sync with `push_tests.yml`'s.**
- `push_tests.yml` (Fast GPU Tests on **main**) gates on `src/diffusers/**.py`, `examples/**.py`, **`tests/**.py`**.
- `pr_tests_gpu.yml` (Fast GPU Tests on **PR**) gates only on a narrow list of core loader/modeling files + `examples/**/*.py`. It **excludes** `src/diffusers/models/**` and `tests/**`.
So a PR that adds a new model + its tests (touching only `src/diffusers/models/**` and `tests/models/**`, as #14181 did) never triggers GPU CI on the PR. The tests first execute post-merge on `main`, or accidentally when some unrelated PR trips the narrow filter.
Net: GPU-only tests were introduced in a PR that structurally couldn't run them.
## Suggested fixes (Claude)
1. **Align the PR trigger with the push trigger** — add `src/diffusers/models/**.py`, `src/diffusers/pipelines/**.py`, `tests/models/**.py`, `tests/pipelines/**.py` to `pr_tests_gpu.yml`'s `paths:`.
2. **Scope the pytest run to changed files** so broadening the trigger doesn't run the entire `tests/models` suite (~17 min) on every model PR. Derive changed test files from `git diff origin/main...HEAD` and pass those (still combined with the mixin `-k` pattern). *This overlaps directly with the "fetch tests to run per PR" work.*
3. **Stopgap:** until the above lands, run `/diffusers-bot pytest ` (via `pr_comment_gpu_tests.yml`) on any new-model / new-transformer PR, and make it a review-checklist item.
@DN6 mentioning you since 2. above is probably also best implemented after your take on improving the test fetcher?
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with .github/workflows/pr_tests_gpu.yml and compare its paths filter with push_tests.yml. Review how changed files could be derived from git diff origin/main...HEAD and how the existing pytest mixin -k pattern is used. Done means new model and test changes trigger PR GPU CI without unnecessarily running the full tests/models suite, with the Cosmos3 case covered.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- github-actions, python, pytorch
- Ambito
- ci-cd, testing-qa
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100