InsightSoftwareConsortium / InsightSoftwareConsortium/ITK

ENH: Remaining follow-ups to PEP 562 lazy loading (#6183)

Open
#6,728 0 comments 0 reactions 0 assignees View on GitHub
area:Python wrapping type:Enhancement
Dominant language
C++
Stars
1.7k
Forks
748
Avg merge
1d 1h
Merged PRs (30d)
64

Description

#6183 landed the PEP 562 baseline for `itk/__init__.py` and the `itk.` synthetic submodules. Of the seven follow-ups identified at the time, three have since shipped (per-module load locks, `ITK_EAGER_IMPORT`, the `-X importtime` benchmark). **Four remain**, listed below in priority order; they are independent and none blocks the others.

Already shipped — for the record, do not re-file

- **Global load lock** — resolved by replacing the single module-level `RLock` with a per-SWIG-module lock map (`itk/__init__.py:62-83`, `_module_load_locks` / `_get_module_load_lock`). First-touch loads of *unrelated* SWIG modules now proceed in parallel from `threading.Pool` / joblib / Dask workers, which was the point of the original request.
- **`ITK_EAGER_IMPORT` escape hatch** — implemented (`itk/__init__.py:223,233`), following the SPEC 1 `EAGER_IMPORT` convention. Restores the debug workflow ("force-load everything and see which submodule fails") that was lost when `itkConfig.LazyLoading` was deprecated.
- **Cold-start benchmark** — `Wrapping/Generators/Python/Tests/lazy_importtime.py`, registered in `Wrapping/Generators/Python/Tests/CMakeLists.txt:116`. Measures `python -X importtime -c 'import itk'` and the post-import `sys.modules` count, and asserts a budget.

Gap 1 (highest user impact) — generate .pyi stubs for itk and itk.<Module>

Currently `git ls-tree -r --name-only main | grep '\.pyi$'` returns **nothing**.

Without stubs, Pyright / Pylance / mypy / VS Code see only `def __getattr__(name): ...` and lose autocomplete for on the order of 10 000 ITK template instantiations. Scientific-Python SPEC 1 cites exactly this as the motivation for `lazy_loader.attach_stub()`.

The wrapping generator (`Wrapping/Generators/Python/igenerator.py`) already computes the per-template name list, so the work is a parallel `.pyi` writer that re-exports those names, plus build-system glue to install them alongside the package. scikit-image's `_skimage/__init__.pyi` is the reference shape.

Estimated ~300-500 lines in the generator plus CMake install rules. Largest single item here and the biggest user-visible improvement.

Gap 2 — document the PEP 810 forward path in the ITK 6 migration guide

PEP 810 (explicit lazy imports) was accepted 2025-11-03 and ships in CPython 3.15. `Documentation/docs/migration_guides/itk_6_migration_guide.md` has a "Python lazy loading is always on" section (around line 411) that documents the removals — `itkConfig.LazyLoading`, `ITK_PYTHON_LAZYLOADING`, `LazyITKModule` — but says nothing about what comes next.

One paragraph is enough: on Python 3.15+ users may write `lazy import itk.ITKCommon`, and the existing PEP 562 mechanism keeps working unchanged because `_make_itk_lazy_submodule` registers each submodule in `sys.modules`. Roughly 15 lines of prose; can ride along with any future ITK 6 documentation PR.

(Today the string "PEP 810" appears exactly once in the tree, in a docstring in `Tests/lazy_importtime.py:35`.)

Gap 3 — first-touch SWIG cost (belongs to the wrapping generator, not to lazy loading)

When `itk.Image` is first touched, the entire `ITKCommon` SWIG `.so` loads, pulling in hundreds of templates. Lazy loading defers that cost; it does not shrink it. Reducing it means either splitting `_module.so` per template family or adopting SWIG's `-builtin` mode — both substantial changes to the wrapping pipeline.

This should be tracked as a separate wrapping-generator design discussion rather than as lazy-loading work; it runs in parallel and does not compete.

Gap 4 — decide on tab-completion forcing a full load

IPython / Jupyter `itk.` walks `__dir__()` and then `getattr()`s each advertised name, which triggers loading every SWIG module — exactly the cost lazy loading exists to avoid. This is a known ecosystem issue: [lazy-loader#155](https://github.com/scientific-python/lazy-loader/issues/155).

Two options: distinguish "advertise" from "resolve" by returning a sentinel proxy from `__getattr__` during completion-time queries (hard, ~50 lines, fragile), or document the behaviour and accept it (zero code). A decision either way closes this item.

Suggested sequencing and references

Order: gap 4 (decide — cheap), gap 2 (doc paragraph), gap 3 (open as a separate wrapping-generator issue), gap 1 (scope and assign; multi-week, single developer).

References:
- #6183 — the baseline (merged)
- [PEP 562 — Module `__getattr__` and `__dir__`](https://peps.python.org/pep-0562/)
- [PEP 810 — Explicit lazy imports](https://peps.python.org/pep-0810/) (accepted 2025-11-03, CPython 3.15)
- [Scientific Python SPEC 1 — Lazy Loading of Submodules and Functions](https://scientific-python.org/specs/spec-0001/)
- [scientific-python/lazy-loader](https://github.com/scientific-python/lazy-loader) — reference implementation ITK mirrors but does not depend on
- [lazy-loader#155](https://github.com/scientific-python/lazy-loader/issues/155) — the tab-completion gotcha

Source touchpoints: `Wrapping/Generators/Python/itk/__init__.py`, `Wrapping/Generators/Python/itk/support/_lazy_submodule.py`, `Wrapping/Generators/Python/igenerator.py`, `Wrapping/Generators/Python/Tests/lazy.py`, `Wrapping/Generators/Python/Tests/lazy_importtime.py`.

Contributor guide

Open the contributing guide

Research direction

Choose one independent gap before starting. Read the named touchpoints in Wrapping/Generators/Python, especially itk/__init__.py, _lazy_submodule.py, igenerator.py, the lazy tests, and the ITK 6 migration guide; use the cited references for context. Done means the selected stub, documentation, generator, or completion behavior has a defined outcome and its relevant tests or installation checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, python
Domain
build-system, developer-experience, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.