Shopify / Shopify/liquid-spec

Provide language tests in liquid repo itself

Open
#37 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
19
Forks
8
Avg merge
6h 41m
Merged PRs (30d)
2

Description

Problem

Generating tests from liquid is a flawed approach, which implicitly couples to those tests following a set of rules to allow the tests to be generated properly. This results in issues like https://github.com/Shopify/liquid-spec/issues/30 and https://github.com/Shopify/liquid-spec/issues/31 and plenty of updates to the upstream liquid repo to get the right test coverage.

This test generation approach is also quite inconvenient, since it means that adding a test requires changes to upstream liquid to add the test as well as a PR to liquid-spec to regenerate the tests. This also means that these two can easily get out of sync, without CI giving feedback of problems from changes to liquid that could break test generation.

The test generation approach also relies on cloning the liquid repo into tmp/liquid and patching it, leading to issues like https://github.com/Shopify/liquid-spec/issues/33 and requires hacks to test against a liquid branch (e.g. to use the test immediately in liquid-wasm).

Proposal

Have the end goal be to have an in-repo gem in the upstream liquid repo with yaml liquid language tests, along with the code to run those tests as part of its test suite. This would only includes tests that don't rely on language extensions (e.g. custom drops, tags or filters) which aren't really testing the core language itself, which could continue to be tested in test/integration folder.

As part of that transition, the liquid tests can continue to be refactored to improve test generation, which will also simplify the final conversion of those ruby tests to YAML tests. For instance, that can include

  • Converting more language tests to use tests helpers decoupled from the liquid library API, such as assert_template_result or assert_match_syntax_error
  • Separating language tests from library tests, such as by not using those decoupled test helpers assert_template_result or assert_match_syntax_error in those library tests
  • Splitting assertions into separate test methods so they can each have a unique and descriptive name, without needing a digest hash for uniqueness

so that we can get to the point where the generated YAML tests could then be committed into the liquid repo and the ruby tests/assertions they were generated from removed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the proposal and inspecting the existing generated tests, the test/integration folder, and the Ruby test helpers mentioned in the issue, including assert_template_result and assert_match_syntax_error. Map which tests cover the core language versus extensions, then confirm the desired end state: YAML language tests and their runner live in the upstream liquid repository without the current generation workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby, yaml
Domain
testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.