WordPress / WordPress/php-toolkit

E2E tests for the entire Blueprint execution flow

Open
#108 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
61
Forks
21
Avg merge
19h 34m
Merged PRs (30d)
7

Description

TL;DR I am drawn to a conclusion that e2e testing for all input types will make test development and runs longer - with little impact on quality. I'd rather focus on better unit testing, as per observations, below.

With a lot of test development done in #82 it is time to garther some thoughts on how to meaningully test this lib.

First of all: #82 has been a constant source of new issues, like #104 #107 or #83 which means e2e really do impact the quality of our work here. It's a trivial conclusion but still - nice to confirm.

Next: I've started with a pipeline running parameterized tests for all types of inputs. This was quickly reduced by one, when we decided to drop BlueprintBuilder as an input type with #98 With this done, I took a moment to examine the main process flow and see how it could be tested best.

  1. The process is initiated with the run_blueprint function; the DI, runtine and env are set up first.
  2. The input is preprocessed with parseAndCompile:
    • Parsing depends on the type of the input:
      • for a JSON string it is decoded and validated. and mapped to a Blueprint,
      • for a stdClass it is validated and also mapped to a Blueprint,
      • and a Blueprint is just validated.
    • Compiling always takes a Blueprint and returns a CompiledBlueprint. At this point all paths have allready converged.
  3. The CompiledBlueprint is run by the BlueprintRunner. Each run produces a result which is an array of StepSuccess or BlueprintRunnerException. At this point it time, the return from the runner is not transformed in any way or form by the engine or the most external run_blueprint function.

Observations:

  • Both validation and mapping are identical for a JSON string and stdClass.
  • Decoding is done trivially with json_decode so there is nothing to trully test.
  • Mapping is done via the BlueprintMapper I wrote some time ago. I'd gladly test it more thoroughly but this can be done via unit tests.
  • Validation is done identically for all types, with a small caveat for the Blueprint type. This is a work-around so it is likely to be rewritten. But either way it can be completely tested with unit tests.
  • For top notch quality parseAndCompile could be tested as a whole subprocess. Those test will be quick, so there is little worry for test execution times if they are run for all input types.
  • What actually makes E2E tests run long is the downloading and installing of resources. So doing it trice mostly triples the pipeline execution flow, with no return on investment.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names run_blueprint, parseAndCompile, BlueprintMapper, and BlueprintRunner. Read those entry points and the existing tests first; determine which subprocess or integration coverage is intended for the converged flow and input validation. Done would require an explicit test scope and passing coverage, but the issue does not currently define that scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.