Comfy-Org / Comfy-Org/Comfy-Desktop
ci: run @lifecycle E2E tests on GitHub runners
- Dominant language
- TypeScript
- Stars
- 458
- Forks
- 59
- Avg merge
- 22h 18m
- Merged PRs (30d)
- 45
Description
## Summary
The `@lifecycle` E2E suite in `e2e/lifecycle.test.ts` (install → detail → launch → console → stop → update) currently runs only on developer machines via:
```sh
pnpm run build && pnpm run test:e2e:windows -- --project=lifecycle
```
It is **not** invoked by `.github/workflows/ci.yml`. The three CI jobs run `test:e2e:macos|windows|linux`, and each Playwright project filters by `grep: /@macos|@windows|@linux/`, so nothing tagged purely `@lifecycle` is picked up.
This suite is the only end-to-end coverage we have for the most user-visible paths (install wizard, launch, console, stop, update). It should run on CI, at minimum on PRs that touch installation/launch/update code paths.
## Why this is feasible on hosted runners
Objections from earlier discussions don't actually hold up:
- **Runtime** — no individual test takes 10 minutes; the 600s project timeout is just a defensive ceiling. Real wall-time is dominated by the standalone download.
- **GitHub API rate-limits** — `lifecycle.test.ts` reads `githubtoken.txt` walking up from `__dirname` purely to authenticate API calls and avoid IP-based rate limits. On hosted runners we either:
- inject `GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}` into the job env (the test already falls back to `process.env['GITHUB_TOKEN']`), or
- skip authentication entirely — GitHub-hosted runners don't generally hit the unauthenticated rate limit for our usage.
- **Disk/network** — ~500 MB CPU standalone download is a non-issue on GitHub-hosted runners.
- **GPU** — the test selects the **CPU** variant (`.variant-card`, `hasText: /CPU/i`), so no GPU is required. We just need to verify the launched ComfyUI server reaches its localhost URL, which is what the test already asserts.
## Proposed work
1. Add `"test:e2e:lifecycle": "playwright test --project=lifecycle"` to `package.json`.
2. Add a new job (or matrix entry) in `.github/workflows/ci.yml` that runs `pnpm run test:e2e:lifecycle`.
- Start on Windows only (matches our existing dev workflow); add macOS/Linux once the Windows run is stable.
- Gate behind `paths-filter` for `src/main/**`, `src/renderer/**`, `e2e/**`, `package.json`, or trigger via `workflow_dispatch` so we don't pay the cost on every doc-only PR.
3. Set `GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}` in the job env so the test's existing token-discovery path picks it up (no test changes required).
4. Verify each of the existing tests passes on the runner before promoting to a required check.
5. Once green, mark the job as required for PRs touching install/launch/update modules.
## Out of scope
- Refactoring `lifecycle.test.ts` itself — the suite is currently passing locally and shouldn't need changes beyond the env-var pickup it already supports.
- Adding new lifecycle scenarios — that can land in follow-ups once the harness is in CI.
## Acceptance criteria
- [ ] CI runs `--project=lifecycle` on at least one platform (Windows) on PRs that touch install/launch/update code paths.
- [ ] The job authenticates GitHub API calls via `secrets.GITHUB_TOKEN` (no `githubtoken.txt` checked in).
- [ ] All existing `@lifecycle` tests pass green on the runner without code changes to the tests.
- [ ] Job time budget documented in the workflow file.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.