Cricket should support testing on tox
- Dominant language
- Python
- Stars
- 226
- Forks
- 77
- Avg merge
- 3h 53m
- Merged PRs (30d)
- 2
Description
tox is heavily used in many major projects for the ease of testing multiple Python configurations, versions of packages, and the like. Cricket should support using tox, because then it's usefulness is amplified in projects that use tox, as it can put a nice face on running lots of different tests on different tox envs from one spot.
Things that need to be taken into consideration:
- Test discovery will have to be per-environment. Test suites do not stay the same on 2.6, 2.7, 3.3, etc, especially in the case of Twisted. We have entire test files that are not on 3.3, some test classes are deleted because they can't be run, and therefore won't show up on test discovery for all platforms. We also generate some test classes depending on what reactors are available, and not all Pythons have the same reactors available. Therefore, during startup, a "discovery" run will have to happen, possibly in a env called something like "py27-tests-discovery" for the "py27-tests" env, which just executes the test runner in a "dry run" configuration (`-n` for Twisted Trial).
- Cricket can't just access the raw unittest results -- it'll have to use something like subunit, which is a machine-readable streaming output format. Trial has built-in support for subunit, and there is a py.test plugin available for it. So, cricket will have to understand this subunit output, but that shouldn't be too difficult.
- Cricket will have to execute tox itself, and have some way of knowing when the tests start and what's tox's output. Tox has `--result-json` but I have no idea if it is streaming (unlikely). Running tests in parallel environments may be desirable, but it may also not be.
Contributor guide
Research direction
Start by examining Cricket's existing test-runner integration and how it could invoke tox. The issue identifies per-environment discovery, subunit output, and tox execution as requirements; define completion around reliably discovering and reporting tests across tox environments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100