exercism / exercism/shen

Build Test Runner

Open
#19 0 comments 0 reactions 0 assignees View on GitHub
x:action/create x:knowledge/advanced x:module/test-runner x:size/massive x:type/coding
Dominant language
Shell
Stars
1
Forks
6
PR merge metrics
No merged PRs in 30d

Description

_This issue is part of the migration to v3. You can read full details about the various changes [here](https://github.com/exercism/v3-launch/)._

In Exercism v3, one of the biggest changes is that we'll automatically check if a submitted solution passes all the tests.

We'll check this via a new, track-specific tool: the [Test Runner](https://github.com/exercism/v3-docs/tree/main/anatomy/track-tooling/test-runners). Each test runner is track-specific. When a new solution is submitted, we run the track's test runner, which outputs a JSON file that describes the test results.

The test runner must be able to run the tests suites of both [Concept Exercises](https://github.com/exercism/v3-docs/blob/main/product/concept-exercises.md) and [Practice Exercises](https://github.com/exercism/v3-docs/blob/main/product/practice-exercises.md). Depending on the test runner implementation, this could mean having to update the Practice Exercises to the format expected by the test runner.

## Goal

Build a test runner for your track according to [the spec](https://github.com/exercism/v3-docs/blob/main/anatomy/track-tooling/test-runners/interface.md).

If you are building a test runner from scratch, we have a [starting guide](https://github.com/exercism/v3-docs/blob/main/anatomy/track-tooling/test-runners/creating-from-scratch.md) and a [generic test runner](https://github.com/exercism/generic-test-runner/) that can be used as the base for the new test runner.

If a test runner has already been built for this track, please check if it works on both Concept Exercises _and_ Practice Exercises.

It can be very useful to check how [other tracks have implemented their test runner](https://github.com/search?q=org%3Aexercism+test-runner&type=repositories).

## Tracking

https://github.com/exercism/v3-launch/issues/4

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the test-runner interface spec and the creating-from-scratch guide, then compare the generic test runner with implementations in other tracks. Check the Shen track's Concept and Practice Exercises and confirm the runner produces the required JSON test results for both; update exercise formats if needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
testing, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.