Collecting tests from package `app` gets broken after adding pytest plugin `app.tests.foo`

Minimal example: after adding app.tests.foo to pytest_plugins in conftest.py, collecting tests fails:

$ find .
.
./conftest.py
./app
./app/tests
./app/tests/foo.py
./app/tests/app
./app/tests/app/test_app.py
./app/tests/app/__init__.py
./app/tests/app2
./app/tests/app2/test_app2.py
./app/tests/app2/__init__.py
./app/src

$ cat conftest.py 
pytest_plugins = ["app.tests.foo"]
$ cat ./app/tests/foo.py
$ cat ./app/tests/app/test_app.py
def test_sum():
    assert 2 + 2 == 4
$ cat ./app/tests/app2/test_app2.py
def test_sum():
    assert 2 + 2 == 4

$ pip freeze
attrs==21.2.0
iniconfig==1.1.1
packaging==21.0
pluggy==1.0.0
py==1.10.0
pyparsing==2.4.7
pytest==6.2.5
toml==0.10.2


$ pytest .
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/q/dev/repro
collected 1 item / 1 error                                                                                                                                                                                                                                                               

========================================================================================================================================= ERRORS =========================================================================================================================================
_______________________________________________________________________________________________________________________ ERROR collecting app/tests/app/test_app.py _______________________________________________________________________________________________________________________
ImportError while importing test module '/home/q/dev/repro/app/tests/app/test_app.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'app.test_app'
================================================================================================================================ short test summary info =================================================================================================================================
ERROR app/tests/app/test_app.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================================================================== 1 error in 0.06s ====================================================================================================================================

Without pytest_plugins = ["app.tests.foo"] everything works fine.

Reproduce the failure using conftest.py, app/tests/foo.py, and the two test modules shown in the minimal example. Run pytest . with and without pytest_plugins = ["app.tests.foo"], then trace the collection/import error involving app/tests/app/test_app.py. Done means enabling the plugin no longer breaks collection and both example tests pass.