lablup / lablup/backend.ai-webui
Reduce E2E skip ratio (38.6%) — triage and remediate 248 skipped tests
- Dominant language
- TypeScript
- Stars
- 133
- Forks
- 81
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 344
Description
## Context
Recent full E2E run statistics:
| Metric | Count |
|---|---|
|—|—|
|---|---|
| Total | 642 |
|---|---|
| Passed | 380 |
|---|---|
| Failed | 14 |
|---|---|
| **Skipped** | **248 (38.6%)** |
|---|---|
| Duration | 39.2 min |
|---|---|
Nearly 4 of every 10 E2E tests are skipped. That is too high to give meaningful coverage signal: a green run can hide product regressions because the affected tests were never executed. We need to triage the skipped pool, classify each test by **why** it skips, and have an explicit remediation path per category.
## Goal
For every skipped test, attribute the skip to one of a small, well-defined set of categories. Then for each category, decide and track the remediation (un-skip / refresh baseline / delete / version-gate properly).
## Hypothesized skip categories (to be confirmed during triage)
1. **`test.fixme(...)` for removed/restructured UI** — the route or component the test depends on was removed or renamed (e.g. `/import` returns 404, `/service/start` removed in FR-2675/FR-2822, `ServiceLauncherCreatePage` deleted). Remediation: either rewrite the test against the new UI flow or delete the test along with the removed feature spec.
1. **`test.fixme(...)` for stale visual-regression baselines** — snapshot diverges because the UI legitimately changed but baselines were never refreshed. Remediation: run `--update-snapshots` against the current build and commit refreshed PNGs in a dedicated PR (do not bundle with functional fixes).
1. **`test.skip(...)` because of missing feature gate** — the test depends on a feature that may or may not be present in the server build (e.g. `ClusterModeFormItems` from FR-2381 requires WebUI > v26.3.0). The test gracefully self-skips when the warning element is absent. Remediation: replace the in-test runtime detection with an explicit version/feature gate (`test.skip()` with a version check, or split into a separate spec tagged with the minimum supported version).
1. **`test.describe.serial` cascade skips** — when an earlier test in a serial block fails, all later tests in the block report as "did not run" / skipped. Counted here as skip pollution from elsewhere in the suite. Remediation: investigate whether serial coupling is genuinely required; split into independent tests where possible so a single failure does not silently mask coverage of N other behaviors.
1. **Explicit `test.skip` for environment-only constraints** — e.g. tests gated on cluster size > 1 when the test backend enforces `aria-valuemax=\"1\"`. Remediation: document the environment requirement in the test name/tag (`@requires-multi-node`), and either set up a matching test backend or move the test to a separate suite that only runs on capable backends.
## Deliverables
- [ ] Inventory of all 248 skipped tests with `file:line`, skip reason quoted from the source, and assigned category.
- [ ] Numeric breakdown by category, surfaced as a chart in the issue comments.
- [ ] One follow-up Jira ticket per category with concrete actions (e.g., \"Refresh visual-regression baselines for the X spec files affected by FR-2664 / FR-2381\").
- [ ] A target skip ratio post-remediation (proposal: ≤ 15%) and a date by which to re-measure.
## Acceptance criteria
- Every currently skipped test has either been (a) re-enabled, (b) deleted along with its removed feature, or (c) gated explicitly with a documented reason tied to a tracking ticket. No \"bare\" `test.fixme(...)` left in the suite.
- Full-run skip ratio drops to the agreed target.
- CI green runs are meaningful again: a passing full E2E suite implies coverage of the entire scoped feature surface, not 60% of it.
## Out of scope
- The 14 currently failing tests. Those are tracked separately as pre-existing infrastructure/timing issues (e.g. `resource-policy.spec.ts` row visibility, `modifyThemeJson` `Failed to fetch`, RBAC dropdown timing) and should be addressed in their own tickets.
—
Captured while working on branch: worktree-FR-3005
JIRA Issue: FR-3109
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.