python / python/cpython

`PyImport_CreateModuleFromInitfunc()`: wrong `__name__` for submodules, non-ASCII names rejected, inittab name clashes

Ouverte
#157,384 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

interpreter-core topic-C-API type-bug
Langage dominant
Python
Étoiles
77.2k
Forks
35.9k
Métriques de merge des PR
Métriques de PR en attente

Description

Bug report

Bug description:

Since PyImport_CreateModuleFromInitfunc() was added (in 3.15, gh-116146), I ran into some issues from real world use internally at Meta (with 3.15+, as well as our 3.14 fork, which has this backported). Collected here as a single issue for convenience - let me know if splitting is preferred.

The API is built on the internal create_builtin() helper, and inherits a few inittab behaviors that don't make sense for an explicitly-passed init function.

  1. Single-phase submodules get the short name. The package context is never set while initfunc runs, so a single-phase init that creates its module as sub (what pybind11's PYBIND11_MODULE(sub, m) does) ends up with __name__ == 'sub' while the spec says pkg.sub. The original impl handled that, but it was lost somewhere on the way.
  2. Non-ASCII spec names raise UnicodeEncodeError, even for multi-phase modules, which the dynamic loader accepts.
  3. Names registered in PyImport_Inittab are handled silently. Depending on the state of the builtin, initfunc is ignored, or the module created here replaces the builtin for later imports, or (multi-phase builtin) initfunc's module overwrites the real one in sys.modules. sys and builtins fall in the first bucket.
  4. Undocumented semantics. Single-phase modules are added to sys.modules by the call itself, multi-phase ones aren't. The spec name is the module's identity, so a second call with the same name and a different initfunc returns the cached module and never calls the new function.

The existing test only covers one top-level single-phase and one top-level multi-phase module, so none of this was caught.

AI disclosure: I ran into the first issue in production, used Claude Opus 5 and Fable 5.1 to root cause, which flagged the other issues while investigating. Fable 5.1 prepared the fixes.

Demo output before any fix:

pkg.sub: repr=<module 'sub'> __name__=sub
non-ASCII name: UnicodeEncodeError: 'ascii' codec can't encode character '\xf6'
same name, initfunc B after A: which=A  calls_a=1 calls_b=0
'_random' via custom initfunc, then `import _random`: <module 'same'>  hasattr(Random)=False

Tests demonstrating all of the above: https://github.com/itamaro/cpython/tree/gh-116146-initfunc-tests-only

Plan

One PR each:

  • Set the package context while calling initfunc, like the dynamic loader does (1)
  • Accept non-ASCII names for multi-phase init; single-phase gets the same SystemError as dynamic loading (2)
  • Raise ImportError for names registered in PyImport_Inittab (3)
  • Document the sys.modules and name-identity behavior (4)
CPython versions tested on:

CPython main branch, 3.15

Operating systems tested on:

Linux, macOS

Linked PRs
  • gh-157387
  • gh-157388
  • gh-157389
  • gh-157390
  • gh-157758

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez par PyImport_CreateModuleFromInitfunc(), les tests existants pour les modules de premier niveau à phase unique et à phases multiples, ainsi que la branche de test liée gh-116146-initfunc-tests-only. Comparez son comportement avec PyImport_Inittab et le chargeur dynamique. Le travail est considéré comme terminé lorsque les quatre éléments de la liste de contrôle sont traités, y compris la sémantique documentée de sys.modules et de l’identité des noms ; les PR liées gh-157387 à gh-157390 et gh-157758 montrent que le travail est déjà en cours.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
backend
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.