abcxyz / abcxyz/abc

Add a `golden-test check` command to run in PRs

Open
#450 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
16
Forks
4
PR merge metrics
No merged PRs in 30d

Description

User journey: as a template author who maintains templates outside of the abc repo, I want an easy way to test my templates as part of the pull request process.

We could create a single command that would be invoked in an`on: pull_request` GitHub Action. This command would:

- Make sure all templates have at least one golden test. If not, print a helpful message about using the `new-test` command which will be released soon.
- Make sure all golden tests are passing (`verify`). If they're failing due to a test data mismatch, suggest running the `record` command. If they're failing due to a broken spec.yaml, identify the failing spec.yaml and describe the problem with it.

All problems should be reported, not just the first problem that is found.

The command could be `abc templates golden-test check`. It could have an optional parameter giving the starting directory.

We could also provide a github action that would download the latest build of `abc` and run this new command.

A good first customer for this would be the github.com/abcxyz/guardian repo. @verbanicm @gjonathanhong

EDIT:

Here's another possibility: expand the scope of this from just golden-tests to everything having to do with abc templates. We could have a command `abc templates check-clean` that would perform all the above checks, and also look for upgrade operations with incomplete merges (there should be no files with an `.abcmerge_...` suffix).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.