python / python/mypy

stubgen doesn't find c modules in namespace of python package

Open
#12,160 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-stubgen
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Bug Report

stubgen does not find c extension modules that are placed in the namespace of a "regular" python package.

More Details:
For "regular" python packages and modules stubgen is using pkgutil.walk_packages() here to find subpackages/submodules but is not checking the namespace so it can miss some.

To Reproduce

This is how I encountered the issue (there is probably a simpler way to reproduce but I'm using gdb here as an example)

(Write your steps here:)

  1. get gdb with python3 support (install or download a statically compiled version)
  2. prepare a little wrapper script to start stubgen:
import sys
import mypy

mypy.util.check_python_version("stubgen")

options = mypy.stubgen.Options(
  pyversion=sys.version_info[:2],
  no_import=False,
  doc_dir="",
  search_path="",
  interpreter="",
  ignore_errors=False,
  include_private=False,
  output_dir="out",
  modules=[],
  packages=["gdb"],
  files=[],
  verbose=False,
  quiet=False,
  export_less=False
)

mypy.stubgen.generate_stubs(options)
  1. start gdb and then source the previously written script: source /path/to/script.py

(these steps are necessary because the gdb python package is only accessible through the python interpreter inside GDB)

Expected Behavior

I was expecting the pyi files to contain all objects in the gdb namespace (including the c modules that gdb puts in the gdb namespace).

Actual Behavior

E.g. gdb.Architecture (and all the other c modules) are not included (can be quickly checked by running grep -r "Architecture" inside out dir) even though they are easily accessible from withing gdb: python help(gdb.Architecture).

Your Environment

I tried it in several environments before investigating, e.g.:

  • Mypy version used: both mypy 0.761 and master (at commit 129dba468c235fbdc01f608736554061368bcff3)
  • Mypy configuration options from mypy.ini (and other config files): I used a clean env without additional configs
  • Python version used: Python 3.8.10
  • Operating system and version: Ubuntu 20.04.3 LTS x86_64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mypy/moduleinspect.py at the pkgutil.walk_packages() logic described in the issue, then trace how stubgen discovers regular package contents. Reproduce with the provided GDB wrapper and check the generated stubs for gdb.Architecture and other C modules. Done means C extension modules exposed in the package namespace appear in the generated .pyi files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.