google / google/site-kit-wp

Implement testing infrastructure for Gutenberg integrations

Open
#4,108 17 comments 1 reaction 0 assignees View on GitHub
P1 Type: Infrastructure
Dominant language
JavaScript
Stars
1.4k
Forks
383
Avg merge
4d 14h
Merged PRs (30d)
77

Description

Based on #4107, this issue is focused on setting up the necessary infrastructure to test Gutenberg integrations as reliably as possible.

Some particular challenges to work around here include:
* Gutenberg has been included in core since WordPress 5.0. Technically even older versions could use the Gutenberg plugin though, which makes things more complicated. Or even any current WordPress version can use the Gutenberg plugin in an even newer version to get more of the "bleeding-edge" changes.
* Gutenberg APIs have been modified over time and unfortunately are by far not as stable/backward-compatible as WordPress core's PHP APIs.

We need a testing infrastructure that caters for this. While it is impossible to cover all possible combinations of versions, we should have a good set to rely on. Potentially, we should introduce a subset of our e2e test suite specifically for Gutenberg-based e2e tests, where we would run for a few more scenarios (e.g. one pre-5.0 with Gutenberg plugin, one with 5.0, one with 5.3, one with latest). In addition to e2e tests (which will be the most reliable automatic way to test the integrations), a good additional measure could be to always use the exact versions used in WordPress 5.0 for all the Gutenberg dependencies that Site Kit is using even in Jest tests (at least this part will technically rely on #4107 to be completed first) - to ensure we don't use APIs/functions that were only supported at a later point.

---------------

_Do not alter or remove anything below. The following sections will be managed by moderators only._

## Acceptance criteria

In this issues, "Gutenberg plugin" refers to the WP Gutenberg plugin that can be used to get the "latest" Gutenberg features/version, as referenced above.

* A set of E2E tests should be created (eg. in a folder like `tests/e2e/gutenberg` or similar) that are used for testing Gutenberg integrations (for now this is only the Idea Hub notice)
* These tests should be run against, at minimum:
* WordPress 5.2 (without the Gutenberg plugin enabled)
* WordPress latest (without the Gutenberg plugin enabled)
* WordPress latest (with the Gutenberg plugin enabled)
* The above version "constraints" should ensure that we are not using APIs/features unavailable in earlier/later versions of WP/Gutenberg

## Implementation Brief

To reduce the complexity of the test run code locally and improve local test speed, I think it's best to only run the various WordPress version tests (eg. `5.2` and `latest`) on GitHub Actions/CI. We should run the tests locally on one specific version, but this can be changed locally using existing tools. Testing mostly on `latest` is fine, and the CI tests will cover any APIs not available in `5.2`. After exploring options during the Hackathon, I think this is the least complex and also much-faster-to-run approach.

In terms of testing with/without the Gutenberg plugin installed: I think this is best done by using our existing "enable plugin" approach, eg using `activatePlugin` (see an example of that helper function usage here: https://github.com/google/site-kit-wp/blob/6fb1e91b6b72f5aa029b28e47da00b13a70fba19/tests/e2e/specs/admin-tracking.test.js#L49). We should have a dedicated "Gutenberg test function" or utility that we can wrap a group of tests in that will run them both with and without the Gutenberg plugin. This will ensure the APIs used work in both the oldest, latest, and future versions of Gutenberg.

This means we won't need to create any dedicated E2E folders or configs for Gutenberg tests. This is contrary to the ACs, but I think is a simpler approach that means Gutenberg tests can be part of the existing tests. Otherwise, we essentially need extra environments, complex scripts, and multiple locations for E2E tests.

* Add an E2E utility function that can wrap `it` and/or `describe` blocks in an `it.each` that runs tests both with and without the latest Gutenberg plugin enabled
* Test an existing Gutenberg integration to ensure it behaves as-expected.
* These tests should use the above "Gutenberg on and off" utility function to ensure they're run both with and without Gutenberg

### Test Coverage

* Add existing test coverage; see main section of IB.

## QA Brief

*

## Changelog entry

* N/A

Contributor guide

Open the contributing guide

Research direction

Start with the existing E2E setup and the activatePlugin usage in tests/e2e/specs/admin-tracking.test.js. Identify the current Gutenberg integration tests for the Idea Hub notice and the GitHub Actions configuration that selects WordPress versions. Done means the integration tests run with Gutenberg enabled and disabled, including WordPress 5.2 and latest coverage in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.