Comfy-Org / Comfy-Org/Comfy-Desktop

ci: run @lifecycle E2E tests on GitHub runners

Open
#456 0 comments 0 reactions 0 assignees View on GitHub
ci testing
Dominant language
TypeScript
Stars
458
Forks
59
Avg merge
22h 18m
Merged PRs (30d)
45

Description

## Summary

The `@lifecycle` E2E suite in `e2e/lifecycle.test.ts` (install → detail → launch → console → stop → update) currently runs only on developer machines via:

```sh
pnpm run build && pnpm run test:e2e:windows -- --project=lifecycle
```

It is **not** invoked by `.github/workflows/ci.yml`. The three CI jobs run `test:e2e:macos|windows|linux`, and each Playwright project filters by `grep: /@macos|@windows|@linux/`, so nothing tagged purely `@lifecycle` is picked up.

This suite is the only end-to-end coverage we have for the most user-visible paths (install wizard, launch, console, stop, update). It should run on CI, at minimum on PRs that touch installation/launch/update code paths.

## Why this is feasible on hosted runners

Objections from earlier discussions don't actually hold up:

- **Runtime** — no individual test takes 10 minutes; the 600s project timeout is just a defensive ceiling. Real wall-time is dominated by the standalone download.
- **GitHub API rate-limits** — `lifecycle.test.ts` reads `githubtoken.txt` walking up from `__dirname` purely to authenticate API calls and avoid IP-based rate limits. On hosted runners we either:
- inject `GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}` into the job env (the test already falls back to `process.env['GITHUB_TOKEN']`), or
- skip authentication entirely — GitHub-hosted runners don't generally hit the unauthenticated rate limit for our usage.
- **Disk/network** — ~500 MB CPU standalone download is a non-issue on GitHub-hosted runners.
- **GPU** — the test selects the **CPU** variant (`.variant-card`, `hasText: /CPU/i`), so no GPU is required. We just need to verify the launched ComfyUI server reaches its localhost URL, which is what the test already asserts.

## Proposed work

1. Add `"test:e2e:lifecycle": "playwright test --project=lifecycle"` to `package.json`.
2. Add a new job (or matrix entry) in `.github/workflows/ci.yml` that runs `pnpm run test:e2e:lifecycle`.
- Start on Windows only (matches our existing dev workflow); add macOS/Linux once the Windows run is stable.
- Gate behind `paths-filter` for `src/main/**`, `src/renderer/**`, `e2e/**`, `package.json`, or trigger via `workflow_dispatch` so we don't pay the cost on every doc-only PR.
3. Set `GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}` in the job env so the test's existing token-discovery path picks it up (no test changes required).
4. Verify each of the existing tests passes on the runner before promoting to a required check.
5. Once green, mark the job as required for PRs touching install/launch/update modules.

## Out of scope

- Refactoring `lifecycle.test.ts` itself — the suite is currently passing locally and shouldn't need changes beyond the env-var pickup it already supports.
- Adding new lifecycle scenarios — that can land in follow-ups once the harness is in CI.

## Acceptance criteria

- [ ] CI runs `--project=lifecycle` on at least one platform (Windows) on PRs that touch install/launch/update code paths.
- [ ] The job authenticates GitHub API calls via `secrets.GITHUB_TOKEN` (no `githubtoken.txt` checked in).
- [ ] All existing `@lifecycle` tests pass green on the runner without code changes to the tests.
- [ ] Job time budget documented in the workflow file.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.