pybind / pybind/pybind11-stubgen

Using `KeysView`/`ValuesView`/`ItemsView` across sub modules can result in "can't find/import" error

Open
#190 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Python
Stars
361
Forks
75
PR merge metrics
No merged PRs in 30d

Description

Consider the following example:

#include <functional>
#include <unordered_map>

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>

namespace py = pybind11;

struct MyKeyA
{
  bool operator==(const MyKeyA&) const = default;
};

struct MyKeyB
{
  bool operator==(const MyKeyB&) const = default;
};

template <>
struct std::hash<MyKeyA>
{
  size_t operator()(const MyKeyA& obj) const
  {
    return 0;  // omitted.
  }
};

template <>
struct std::hash<MyKeyB>
{
  size_t operator()(const MyKeyB& obj) const
  {
    return 0;  // omitted.
  }
};

PYBIND11_MODULE(my_native_module, m)
{
  py::module_ module_a = m.def_submodule("module_a");
  py::class_<MyKeyA>(module_a, "MyKeyA");
  py::bind_map<std::unordered_map<MyKeyA, int>>(module_a, "MapAToInt");  // (1)

  py::module_ module_b = m.def_submodule("module_b");
  py::class_<MyKeyB>(module_b, "MyKeyB");
  py::bind_map<std::unordered_map<MyKeyA, bool>>(module_b, "MapAToBool");  // (2)
  py::bind_map<std::unordered_map<MyKeyB, bool>>(module_b, "MapBToBool");
}

The pattern here is:

  • There are two different key types KeyA and KeyB, one defined in sub-module module_a the other in module_b.
  • module_a defines a map dict[KeyA, int] in line (1)
  • module_b defines two maps, dict[KeyA, bool] and dict[KeyB, bool].

Trying to generate stubs for the module fails with:

pybind11_stubgen - [  ERROR] In my_native_module.module_b.MapAToBool.keys : Can't find/import 'my_native_module.module_a.KeysView'
pybind11_stubgen - [   INFO] Terminating due to previous errors

The issue seems to be triggered by the co-existence of lines (1) and (2), i.e., if KeysA gets used in module_a in a map, it somehow "moves the KeysView[...] type over" into that module. By commenting out line (1), the KeysView ends up in the other sub-module which works. Conversely, commenting out line (2) also works, because then module_b is not trying to reference the invalid KeysView path.

For comparison, when commenting out line (1) the symbols in submodule a/b are divided as (trimmed outputs of dir(my_native_module.module_X)):

# Symbols in module_a:
MyKeyA

# Symbols in module_b:
ItemsView[my_native_module.module_a.MyKeyA, bool]
ItemsView[my_native_module.module_b.MyKeyB, bool]
KeysView[my_native_module.module_a.MyKeyA]
KeysView[my_native_module.module_b.MyKeyB]
MapAToBool
MapBToBool
MyKeyB
ValuesView[bool]

Note that all the KeysView are local to b. Commenting in line (1) "moves" the KeysView into module_a leading to the problematic module path:

# Symbols in module_a:
ItemsView[my_native_module.module_a.MyKeyA, int]
KeysView[my_native_module.module_a.MyKeyA]
MapAToInt
MyKeyA
ValuesView[int]

# Symbols in module_b:
ItemsView[my_native_module.module_a.MyKeyA, bool]
ItemsView[my_native_module.module_b.MyKeyB, bool]
KeysView[my_native_module.module_b.MyKeyB]
MapAToBool
MapBToBool
MyKeyB
ValuesView[bool]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided C++ reproduction using two submodules and inspect how generated KeysView, ValuesView, and ItemsView symbols are assigned to module paths. Confirm the failure with pybind11-stubgen; done when the example generates stubs without trying to import KeysView from the wrong submodule.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.