mitmproxy / mitmproxy/pdoc

Force display of method if defined in stub file

Open
#783 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.5k
Forks
228
PR merge metrics
No merged PRs in 30d

Description

I'm currently writing a (company internal) library with one module behaving somewhat similar to python ctypes but im programming it in rust using pyo3. I have implemented fix sized types like U8 for unsigned integer with 8 bits and the possibility to create arrays by using the __get_item__ method, like:

U8(8) # instance of a non array type
U8[3]([0x01, 0x02, 0x03]) # instance of a array type with size 3

Since this is a method on the class but not the instance, I need to define the method on the metaclass. This is currently not supported in pyo3, so instead I load the rust class, and inherit into a new class like this (__init__.py):

from typing import Protocol, Any
from pdoc_example import _rust

class _ArrayAble(Protocol):
    @classmethod
    def __make_array__[C](cls: type[C], size: int) -> Any: ...

U8Array = _rust.U8Array

class _U8ArrayMeta(type):
    def __getitem__[C: _ArrayAble](cls: type[C], size: int) -> type[U8Array]:
        return cls.__make_array__(size)

class U8(_rust.U8, metaclass=_U8ArrayMeta): ... 

Classes that dont need to be changed are just reexported like U8Array.
The import is from a local module named _rust.pyd and stup files for the content are provided under a folder _rust. This way both the type checker and the runtime imports are satisfied.

pdoc does currently not resolve the pyi file for imports like vscode does with the pylance plugin.
For pdoc to properly annotate the types i need to add a __init__.pyi next to the __nit__.py file so my file structure looks like this:

PDOC_EXAMPLE
│ py.typed
│ _rust.cp312-win_amd64.pyd
init.py
init.pyi
└───_rust/__init__.pyi

class U8:
    """U8 in __init__.pyi"""
    def __init__(self, value: int) -> None:
        """U8 init function in __init__.pyi"""
    @property
    def value(self) -> int:
        """U8 property value in __init__.pyi"""
    
class U8Array:
    """U8Array in __init__.pyi"""
    def __init__(self, value: list[int]) -> None:
        """U8Array init function in __init__.pyi"""
    @property
    def value(self) -> list[int]:
        """U8Array property value in __init__.pyi"""

If i run pdoc, the documentation will be correct for U8Array (since it is directly imported) but not for U8 since this is an inherited method and we dont publish those.

Image

I understand the decision to not show inherited members that are not itself documented in the package. I wouldnt even want the information what methods are inherited if i have no possibility to jump to the inherited methods documentation.

However I would like to have a way to document methods that actually are available.
It's also not possible to document class attributes if they are not actually available in the variable but i want the user to set them:

# the user should set the size variable if he inherits from U8Array and i want to tell him that in the documentation
class MyArray(U8Array):
    _size_ = 3

On creation U8Array will look for this classvar and raise an exception if it is not available. You are not supposed to use the U8Array directly but inherit like in the above example or use the U8[size] syntax.

Any documentation inside the pyi file will be dismissed because pdoc will do the dynamic analysis and not allow me to add anything other than annotations for methods and variables that were resolved before.

I'd be willing to contribute a pull request but since there are probably more than one way to solve this issue and it might involve some architectural decisions i first wanted to raise the discussion. Overall my goal is to find a way to satisfy both vscode/pyright and pdoc

  • should pdoc try to enhance import statements from compiled modules with stub files
  • should there be a possibility to document features that are not available or inherited in the dynamic code resolution

I can also provide a mvp setup to reproduce the issues i described if wanted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the documented package layout with init.py, init.pyi, and _rust/init.pyi, then compare pdoc's output for U8 and U8Array. Read the import and dynamic-analysis behavior involved in resolving compiled-module imports and stub files. Done means the documented inherited methods and supported class attributes are represented while runtime and type-checker behavior remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.