shopware / shopware/shopware

Add 6.8 major-feature visual regression coverage for Administration

Open
#19,905 0 comments 0 reactions 0 assignees View on GitHub
domain/quality-ops major/6.8 priority/high testing/visual
Dominant language
PHP
Stars
3.4k
Forks
1.2k
Avg merge
3d 55m
Merged PRs (30d)
436

Description

### User story

## Context

The current visual regression suite runs Playwright tests tagged `@Visual` in the `Visual` project. It exercises the default feature configuration and compares screenshots against committed `*-Visual-linux.png` baselines.

The 6.8 Administration path is activated through `FEATURE_ALL=major` and can render different components and layouts, including `mt-tabs`. Its screenshots must not reuse the default-path baselines: some differences are intentional, while unintended differences need review.

This is the automated follow-up to the manual audit of `mt-tabs` surfaces with `V6_8_0_0` enabled: https://github.com/shopware/shopware/issues/19903

## Goal

Run Administration visual regression tests with `FEATURE_ALL=major` and maintain a separate, reviewable screenshot baseline for the 6.8 path.

## Implementation (TBD)

1. Add a dedicated Playwright project, for example `VisualMajor`, based on the existing `Visual` project:
- Chromium
- `@Visual` grep
- same deterministic viewport and setup
- distinct snapshot suffix/baseline, e.g. `*-VisualMajor-linux.png`

2. Extend the visual-test workflow with a major-feature lane:
- pass `major: true` to the ATS composite action so it exports `FEATURE_ALL=major` before Shopware is installed and built;
- run the `VisualMajor` project with one worker;
- upload its traces, report, and screenshot differences separately from the default `Visual` lane.

3. Create and commit the initial `VisualMajor` baseline snapshots.

4. Ensure the visual suite covers the 6.8-specific `mt-tabs` paths identified by the audit. Add deterministic visual scenarios where current `@Visual` coverage does not reach them, at least:
- Product media modal
- Media folder settings modal
- Flow Builder listing
- Settings → Search → Searchable content

### Acceptance criteria

- A CI workflow can run the complete visual suite with `FEATURE_ALL=major`.
- The 6.8 run uses separate snapshot files from the default feature path.
- Screenshot results and traces are available for review for both lanes.
- Initial baselines are reviewed and committed.
- The identified `mt-tabs` surfaces are covered by deterministic visual tests.
- Existing default-path visual regression testing remains unchanged.

### Definition of Done

- [ ] Fulfills all acceptance criteria defined during discovery.
- [ ] Integration/E2E testing in staging is done.
- [ ] All integration/E2E/unit tests passing; all critical or high-priority bugs are resolved.
- [ ] Fulfills compliance, performance, security, and cloud-readiness needs.
- [ ] Observability—includes monitoring, alerting, and logging; incident response handbooks updated.
- [ ] Documentation—developer docs are written or updated, including information on how the feature or change is adopted and tested. If new functionality, end user documentation is written or updated.
- [ ] (conditional) All new API endpoints have a defined API schema.
- [ ] (conditional) If the change affects the Storefront, it has been tested in the latest three versions of Firefox, Chrome, Edge, IE11, and Safari, as well as the different viewports and devices (Desktop / Tablet / Mobile).
- [ ] (conditional) If the change affects the Administration, it has been tested in current versions of Firefox, Chrome, and Edge.
- [ ] (conditional) Functionality that includes new API routes are callable via the Store-API (/store-api/[version]/).
- [ ] (conditional) If a new Admin module is created, the access control layer is provided, and permissions can be managed via the Users & Permissions module.
- [ ] (conditional) If a new UI input field or interactive element is introduced, the ACL wrapper is added.
- [ ] (conditional) Performance testing done if the change has a high performance impact, such as affecting scalability, load balancing, or query optimization.

Contributor guide

Open the contributing guide

Research direction

Start with the existing Playwright Visual project and @Visual tests, then inspect the visual-test workflow and ATS composite action to trace how configuration and snapshots are handled. Add the VisualMajor lane with FEATURE_ALL=major, separate baseline and artifacts, and deterministic coverage for the Product media modal, Media folder settings modal, Flow Builder listing, and Settings → Search → Searchable content. Done means reviewed initial baselines, separate CI results and traces, and unchanged default-path coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright
Domain
ci-cd, frontend, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.