Discussion: fixing test module name collisions in namespace packages

In #5147 I addressed the issues with pytest when using namespace packages (i.e., directories without a __init__.py file).¹ The problems discussed here are related to that and stem from the same source:

1. If you have namespace packages foo and bar, you cannot currently have foo/util_test.py and bar/util_test.py because these will both cause creation of a module with a fully-qualified name of util_test and insert it (unnecessarily) into sys.modules. (Pytest does catch this collision, for test files anyway, and generates an error during collection.)
2. sys.path has an extra entry added to it for each namespace module, which can cause unexpected changes in what modules get imported as compared to running outside pytest.

A solution for this (for Python ≥3.5, anyway) is already available in the py package: use importlib instead of the older import routines so you can avoid both adding the module to sys.modules and changing sys.path. (This is done by passing ensuresyspath='importlib' to LocalPath.pyimport(); the support for this was added in commit 055bcb1a in 2018.) However, pytest does not currently provide any way to do this, and the best way to enable this is not clear at the moment, at least to me. (One of my worries is issues with backwards compatibility.)

To solve this issue for my projects, I've created this plugin. (Depending on when you're reading this, a more recent version may be available on the master branch.)²

This uses essentially the same technique as py's LocalPath.pyimport() above: use the Python 3.5 importlib API to load a spec, create the module and execute its code, all directly from the file without using sys.path, and without inserting the newly-created module into sys.modules. My version is slightly more complex because I do this only for files matching *.pt (and need to use a couple of tricks to get the interpreter to load files not ending in .py); this preserves backward compatibility for *_test.py and test_*.py files.³

So that's a proof-of-concept. I'd love to have feedback on any of the particulars of how I'm doing that, whether related to the collision/path problem or just the ability to use files not ending in .py.

I'm also hoping we can discuss further how we might move forward with a real fix for the issues raised by namespace packages (for both test modules, as discussed here, and code-under-test modules, as discussed in #5147): the collisions and the sys.path changes. Having done this, it seems to have opened up the possibility of being fixed with a plug-in instead of by changing pytest itself, which I suppose has its advantages and disadvantages.

Thoughts?

¹ Namespace package problems have also been discussed in other issues: #2371, I think #1028, and #221. I suspect that what fixed that last one is the source of our current problems.

² If you are a bit confused about what you see under the src/ subdirectory of that repo, yes, I am indeed using pytest to test my 6502 assembler code. :-) You can see some more "normal" testing of Python code under the lib/ directory.

³ Actually, the main reason I did this was so I can use foo.* on the command line to reference all files related to module foo, rather than foo{.md,_test.py,.py}, especially when there are foobar* files kicking around, too, but the ability to separate the files with the new behaviour and maintain backward-compatibility is a nice bonus.

Start with py/_path/local.py:646 and LocalPath.pyimport(), then compare the linked proof-of-concept conftest.py and the namespace-package context in #5147. Work out a backward-compatible pytest approach for loading these modules without unwanted sys.modules collisions or sys.path changes; done should include an agreed direction and coverage for both namespace-package cases.