beeware / beeware/cricket

Cricket should support testing on tox

Open
#42 2 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
226
Forks
77
Avg merge
3h 53m
Merged PRs (30d)
2

Description

tox is heavily used in many major projects for the ease of testing multiple Python configurations, versions of packages, and the like. Cricket should support using tox, because then it's usefulness is amplified in projects that use tox, as it can put a nice face on running lots of different tests on different tox envs from one spot.

Things that need to be taken into consideration:
- Test discovery will have to be per-environment. Test suites do not stay the same on 2.6, 2.7, 3.3, etc, especially in the case of Twisted. We have entire test files that are not on 3.3, some test classes are deleted because they can't be run, and therefore won't show up on test discovery for all platforms. We also generate some test classes depending on what reactors are available, and not all Pythons have the same reactors available. Therefore, during startup, a "discovery" run will have to happen, possibly in a env called something like "py27-tests-discovery" for the "py27-tests" env, which just executes the test runner in a "dry run" configuration (`-n` for Twisted Trial).
- Cricket can't just access the raw unittest results -- it'll have to use something like subunit, which is a machine-readable streaming output format. Trial has built-in support for subunit, and there is a py.test plugin available for it. So, cricket will have to understand this subunit output, but that shouldn't be too difficult.
- Cricket will have to execute tox itself, and have some way of knowing when the tests start and what's tox's output. Tox has `--result-json` but I have no idea if it is streaming (unlikely). Running tests in parallel environments may be desirable, but it may also not be.

Contributor guide

Open the contributing guide

Research direction

Start by examining Cricket's existing test-runner integration and how it could invoke tox. The issue identifies per-environment discovery, subunit output, and tox execution as requirements; define completion around reliably discovering and reporting tests across tox environments.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.