mitmproxy / mitmproxy/pdoc

Name resolution fails for binary lib +.pyi stubs

Open
#870 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.5k
Forks
228
PR merge metrics
No merged PRs in 30d

Description

Problem Description

In mcumgr-toolkit, I use Rust + PyO3 to compile a binary Python module. I then use pyo3-stub-gen to use my Rust source code to generate a .pyi stubs file.

I then install the code to a virtualenv using pip install. That means, the Python module itself lives in the virtual environment as a binary library file together with a .pyi stubs file.

If I then generate the Python documentation through pdoc, it fails because the types are not declared in order in the .pyi file (which they don't have to be, as far as I know, so this should be an issue on pdoc side).

Warn: Error parsing type stubs for mcumgr_toolkit:
Traceback (most recent call last):
  File "/home/martin/work/zephyr-mcumgr-client/mcumgr-toolkit-python/.venv/lib/python3.12/site-packages/pdoc/doc_pyi.py", line 133, in include_typeinfo_from_stub_files
    imported_stub = _import_stub_file(module.modulename, stub_file)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/martin/work/zephyr-mcumgr-client/mcumgr-toolkit-python/.venv/lib/python3.12/site-packages/pdoc/doc_pyi.py", line 66, in _import_stub_file
    loader.exec_module(m)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/martin/work/zephyr-mcumgr-client/mcumgr-toolkit-python/.venv/lib/python3.12/site-packages/mcumgr_toolkit/__init__.pyi", line 51, in <module>
    class FileChecksumProperties:
  File "/home/martin/work/zephyr-mcumgr-client/mcumgr-toolkit-python/.venv/lib/python3.12/site-packages/mcumgr_toolkit/__init__.pyi", line 56, in FileChecksumProperties
    def format(self) -> FileChecksumDataFormat:
                        ^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'FileChecksumDataFormat' is not defined

If I replace all direct class references with strings (like here def format(self) -> "FileChecksumDataFormat" instead of def format(self) -> FileChecksumDataFormat), then it works fine.

Steps to reproduce the behavior:

I can provide some if required, but it will be kind of ugly since they will contain Rust compilation steps. I'm not sure how to reproduce them without a compiled binary lib.

System Information

pdoc: 16.0.0
Python: 3.12.3
Platform: Linux-6.8.0-100-generic-x86_64-with-glibc2.39

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pdoc/doc_pyi.py, especially include_typeinfo_from_stub_files and _import_stub_file, and reproduce the failure with the reported binary module and .pyi stub setup. The work is done when pdoc can generate documentation from valid stubs whose referenced types are declared later, without requiring those references to be quoted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.