We're encountering some perplexing behaviour regarding Python namespaces when moving to CKAN 2.11.5, and I'm hoping that someone else will have insights.

We're encountering some perplexing behaviour regarding Python namespaces when moving to CKAN 2.11.5, and I'm hoping that someone else will have insights.

We run two sites, with different plugin sets that both include `ckanext-s3filestore`, and one works fine on CKAN 2.11.5, while the other crashes when attempting to run any CLI command on 2.11.5, with a stack trace similar to:

07:56:09,183 INFO [ckan.config.environment] Loading static files from public
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/ckan", line 8, in
sys.exit(ckan())
^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1077, in main
with self.make_context(prog_name, args, **extra) as ctx:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 943, in make_context
self.parse_args(ctx, args)
File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 120, in parse_args
result = super().parse_args(ctx, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1644, in parse_args
rest = super().parse_args(ctx, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 1408, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
value = self.process_value(ctx, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/lib64/python3.11/site-packages/click/core.py", line 2362, in process_value
value = self.callback(ctx, self, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 130, in _init_ckan_config
_add_ctx_object(ctx, value)
File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 139, in _add_ctx_object
ctx.obj = CtxObject(path)
^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/src/ckan/ckan/cli/cli.py", line 56, in __init__
self.app = make_app(raw_config)
^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/src/ckan/ckan/config/middleware/__init__.py", line 29, in make_app
load_environment(conf)
File "/mnt/local_data/ckan_venv/src/ckan/ckan/config/environment.py", line 68, in load_environment
p.load_all()
File "/mnt/local_data/ckan_venv/src/ckan/ckan/plugins/core.py", line 148, in load_all
load(*plugins)
File "/mnt/local_data/ckan_venv/src/ckan/ckan/plugins/core.py", line 164, in load
service = _get_service(plugin)
^^^^^^^^^^^^^^^^^^^^
File "/mnt/local_data/ckan_venv/src/ckan/ckan/plugins/core.py", line 298, in _get_service
return ep.load()(name=plugin_name)
^^^^^^^^^
File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 202, in load
module = import_module(match.group('module'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1126, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ckanext.s3filestore'

Investigation showed various workarounds that would resolve it, but we're still scratching our heads about exactly what is happening here.

- If we undo commit 45572908ac9573aaff8f7db7487b6a112ee6d7e5 and restore the namespace declaration in `ckanext/stats/__init__.py`, then the error goes away.
- Similarly, if we add a namespace declaration to one of our custom plugins, _in a submodule of ckanext_, or install a plugin that has such a declaration, then the error goes away. A namespace declaration in `ckanext/__init__.py` does not suffice. This appears to be why one of our two sites is working, as it installs ckanext-ytp-comments, ckanext-archiver, ckanext-harvest, and ckanext-datarequests, all of which have that declaration.
- If we revert `pysolr` to version 3.9.0 (as was used in CKAN 2.11.4), then the error goes away.

Removing `__init__.py` from ckanext-s3filestore, however, to comply with `importlib` style, does not help.

Does anyone have more insight on what is happening and what is the proper solution to this?

_Originally posted by @ThrawnCA in https://github.com/ckan/ckan/discussions/9340_

Reproduce a failing CKAN 2.11.5 CLI command and compare plugin loading through ckan/cli/cli.py, ckan/config/environment.py, and ckan/plugins/core.py. Then compare the namespace declarations in ckanext/stats/__init__.py and ckanext-s3filestore, plus pysolr 3.9.0 versus the current version; done means identifying and documenting the proper fix for the ModuleNotFoundError.

Written by the indexing model from the issue text.