python-trio / python-trio/trio

Fix our test layout

Open
#274 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pytest-trio relevant todo soon
Dominant language
Python
Stars
7.3k
Forks
431
Avg merge
2d 17h
Merged PRs (30d)
6

Description

Right now our test layout is a mess. Tests are included in the package so that users can run pytest --pyargs trio to make sure it works on their system, but this prevents the use of pytest.ini or configuring extra arguments like --run-slow from conftest.py. We have tests split across trio/tests and trio/_core/tests, and since they're not subdirectories of each other then there's configury that has to be copy-pasted into their appropriate conftest.py files. (And we're not even consistent about this layout, b/c the tests for trio/testing/ are in trio/tests/test_testing.py.) Also these look like public modules, but they aren't really.

The major decision to make is, should we continue distributing tests inside trio? The advantage is that users can (perhaps) test the actual real installed version of trio, and they don't even have to set up some weird dev environment or anything. (Though they do need to install the test requirements, which aren't documented anywhere outside a source tree.) The main disadvantages are (a) we're swimming upstream a bit here, as noted by the issues with pytest.ini and friends, (b) it ~doubles the on-disk size of our installed package for a feature that only a tiny tiny fraction of deployments will use.

(Another commonly cited argument for separating tests out of your package is so that you can run the tests from version A against package version B. This doesn't make any sense to me, since our tests are unit tests that are tightly tied to a specific version of trio. It's not like we're implementing some generic 3rd party spec, and if we were then the compliance tests should be a standalone package anyway...)

How much extra disk space are we talking about? It's small, but less trivial than I was expecting. (I guess that 4 KiB minimum block size adds up.) On my laptop on 2017-08-09, building a wheel from trio master and then installing it in a venv, du reports that the .py and .pyc files take up 1.8 MB total, of which 779 KB is tests. This is not huge in the grand scheme of things, but it's not nothing, either. And in particular in these days of docker deployments, it seems likely that some of our users will be obsessed with making their docker containers smaller and get frustrated and angry about a megabyte of "waste". Maybe this makes sense, maybe not, but it doesn't really matter: making users frustrated and angry is a good thing to avoid no matter why it happens :-).

OTOH twisted does ship their tests inside their source tree, and twisted is massive: 27 MB installed, of which 15 MB is tests. And tornado and asyncio seem to as well. (Though distributors do often take special measures to delete CPython's test suite, which would include asyncio's test suite.)

If we do keep the tests inside trio/, then they should probably all move and be consolidated under trio/_tests/.

If we move them outside the source tree, then we probably have to switch to a layout like tests/ for tests, and src/trio for the actual package. (Explanation.) Which annoys me because it's an extra bit of friction every time you want to open a file, and I find it very handy to run tests or random little scripts directly against the source checkout. Like, tox is cool and all, but it slows things down, and when running locally I want to turn out tests ASAP, in-depth testing is what CI is for.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing test directories trio/tests and trio/_core/tests, along with trio/testing/test_testing.py, pytest.ini, and the relevant conftest.py files. Compare the current package and test layout with the two alternatives described in the issue, then document a decided layout and verify that installed-package testing and local pytest configuration work as intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.