lablup / lablup/backend.ai-webui

Reduce E2E skip ratio (38.6%) — triage and remediate 248 skipped tests

Open
#7,846 0 comments 0 reactions 0 assignees View on GitHub
e2e quick-capture tech-debt
Dominant language
TypeScript
Stars
133
Forks
81
Avg merge
1d 11h
Merged PRs (30d)
344

Description

## Context

Recent full E2E run statistics:

| Metric | Count |
|---|---|

|—|—|
|---|---|

| Total | 642 |
|---|---|

| Passed | 380 |
|---|---|

| Failed | 14 |
|---|---|

| **Skipped** | **248 (38.6%)** |
|---|---|

| Duration | 39.2 min |
|---|---|

Nearly 4 of every 10 E2E tests are skipped. That is too high to give meaningful coverage signal: a green run can hide product regressions because the affected tests were never executed. We need to triage the skipped pool, classify each test by **why** it skips, and have an explicit remediation path per category.

## Goal

For every skipped test, attribute the skip to one of a small, well-defined set of categories. Then for each category, decide and track the remediation (un-skip / refresh baseline / delete / version-gate properly).

## Hypothesized skip categories (to be confirmed during triage)

1. **`test.fixme(...)` for removed/restructured UI** — the route or component the test depends on was removed or renamed (e.g. `/import` returns 404, `/service/start` removed in FR-2675/FR-2822, `ServiceLauncherCreatePage` deleted). Remediation: either rewrite the test against the new UI flow or delete the test along with the removed feature spec.
1. **`test.fixme(...)` for stale visual-regression baselines** — snapshot diverges because the UI legitimately changed but baselines were never refreshed. Remediation: run `--update-snapshots` against the current build and commit refreshed PNGs in a dedicated PR (do not bundle with functional fixes).
1. **`test.skip(...)` because of missing feature gate** — the test depends on a feature that may or may not be present in the server build (e.g. `ClusterModeFormItems` from FR-2381 requires WebUI > v26.3.0). The test gracefully self-skips when the warning element is absent. Remediation: replace the in-test runtime detection with an explicit version/feature gate (`test.skip()` with a version check, or split into a separate spec tagged with the minimum supported version).
1. **`test.describe.serial` cascade skips** — when an earlier test in a serial block fails, all later tests in the block report as "did not run" / skipped. Counted here as skip pollution from elsewhere in the suite. Remediation: investigate whether serial coupling is genuinely required; split into independent tests where possible so a single failure does not silently mask coverage of N other behaviors.
1. **Explicit `test.skip` for environment-only constraints** — e.g. tests gated on cluster size > 1 when the test backend enforces `aria-valuemax=\"1\"`. Remediation: document the environment requirement in the test name/tag (`@requires-multi-node`), and either set up a matching test backend or move the test to a separate suite that only runs on capable backends.

## Deliverables

- [ ] Inventory of all 248 skipped tests with `file:line`, skip reason quoted from the source, and assigned category.
- [ ] Numeric breakdown by category, surfaced as a chart in the issue comments.
- [ ] One follow-up Jira ticket per category with concrete actions (e.g., \"Refresh visual-regression baselines for the X spec files affected by FR-2664 / FR-2381\").
- [ ] A target skip ratio post-remediation (proposal: ≤ 15%) and a date by which to re-measure.

## Acceptance criteria

- Every currently skipped test has either been (a) re-enabled, (b) deleted along with its removed feature, or (c) gated explicitly with a documented reason tied to a tracking ticket. No \"bare\" `test.fixme(...)` left in the suite.
- Full-run skip ratio drops to the agreed target.
- CI green runs are meaningful again: a passing full E2E suite implies coverage of the entire scoped feature surface, not 60% of it.

## Out of scope

- The 14 currently failing tests. Those are tracked separately as pre-existing infrastructure/timing issues (e.g. `resource-policy.spec.ts` row visibility, `modifyThemeJson` `Failed to fetch`, RBAC dropdown timing) and should be addressed in their own tickets.

Captured while working on branch: worktree-FR-3005

JIRA Issue: FR-3109

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.