[BUG]: integral caster inconsistencies

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

3.0.1

Problem description

There are some inconsistencies with type hinting and behaviour of the integral casters from Python to C++.

Type hint history

The type hint was historically int.
#5540 changed the converting type hint to typing.SupportsInt
#5891 changed the converting type hint to typing.SupportsInt | typing.SupportsIndex

The noconvert type hint is still int

Converting behaviour

The converting behaviour accepts anything that implements __index__ or __int__ with the explicit exception of the Python float. numpy.float32 is allowed for example.

If the intention is to only accept real integer arguments only supporting __index__ may be the better option.

Noconvert behaviour

The signed noconvert behaviour will accept any argument that implements __index__.
The unsigned noconvert behaviour will only accept PyLong

If the intention is to only support PyLong (int) as described in the type hint then this line is wrong. It allows the argument if it is a PyLong or it has __index__ which then fails in PyLong_AsUnsignedLong and PyLong_AsUnsignedLongLong which expect PyLong

If we want to support __index__ in noconvert the type hint will need updating (I don't think io_name can support this) and the unsigned caster will need fixing.

Unsigned behaviour

After stepping through the code I better understand what it is doing.
The signed variants call PyLong_AsLong or PyLong_AsLongLong which accept PyLong or SupportsIndex.
The unsigned variants call PyLong_AsUnsignedLong and PyLong_AsUnsignedLongLong which only accept PyLong.
If that fails and noconvert is set it returns false otherwise it falls back to PyNumber_Long which calls __index__ (or __int__) and then calls itself with this new argument.

Code

main caster
unsigned casting

Notes

The casting behaviour uses a mixture of PyLong_AsLong, PyLong_AsLongLong, PyLong_AsUnsignedLong and PyLong_AsUnsignedLongLong depending on if the type is signed and its size.

Since Python 3.8 PyLong_AsLong and PyLong_AsLongLong will use __index__ if available and since Python 3.10 it does not use __int__.

Reproducible example code

m.def("int8_noconvert", [](std::int8_t) { }, py::arg("arg").noconvert());
    m.def("int8_convert", [](std::int8_t) { }, py::arg("arg"));
    m.def("int16_noconvert", [](std::int16_t) { }, py::arg("arg").noconvert());
    m.def("int16_convert", [](std::int16_t) { }, py::arg("arg"));
    m.def("int32_noconvert", [](std::int32_t) { }, py::arg("arg").noconvert());
    m.def("int32_convert", [](std::int32_t) { }, py::arg("arg"));
    m.def("int64_noconvert", [](std::int64_t) { }, py::arg("arg").noconvert());
    m.def("int64_convert", [](std::int64_t) { }, py::arg("arg"));

    m.def("uint8_noconvert", [](std::uint8_t) { }, py::arg("arg").noconvert());
    m.def("uint8_convert", [](std::uint8_t) { }, py::arg("arg"));
    m.def("uint16_noconvert", [](std::uint16_t) { }, py::arg("arg").noconvert());
    m.def("uint16_convert", [](std::uint16_t) { }, py::arg("arg"));
    m.def("uint32_noconvert", [](std::uint32_t) { }, py::arg("arg").noconvert());
    m.def("uint32_convert", [](std::uint32_t) { }, py::arg("arg"));
    m.def("uint64_noconvert", [](std::uint64_t) { }, py::arg("arg").noconvert());
    m.def("uint64_convert", [](std::uint64_t) { }, py::arg("arg"));

This is the current behaviour

import unittest

import numpy

from amulet.level.bedrock.raw_dimension import (
    int8_convert,
    int16_convert,
    int32_convert,
    int64_convert,
    uint8_convert,
    uint16_convert,
    uint32_convert,
    uint64_convert,
    int8_noconvert,
    int16_noconvert,
    int32_noconvert,
    int64_noconvert,
    uint8_noconvert,
    uint16_noconvert,
    uint32_noconvert,
    uint64_noconvert,
)


class MyFloat:
    def __init__(self, value: float) -> None:
        self._value = float(value)

    def __float__(self) -> float:
        return self._value

    def __int__(self) -> int:
        return int(self._value)


class MyIndex:
    def __init__(self, value: int) -> None:
        self._value = value

    def __index__(self) -> int:
        return self._value


class MyInt:
    def __init__(self, value: int) -> None:
        self._value = value

    def __int__(self) -> int:
        return self._value


class IntTestCase(unittest.TestCase):
    def test_int_noconvert(self) -> None:
        for func, signed, convert in (
            (int8_convert, True, True),
            (int16_convert, True, True),
            (int32_convert, True, True),
            (int64_convert, True, True),
            (uint8_convert, False, True),
            (uint16_convert, False, True),
            (uint32_convert, False, True),
            (uint64_convert, False, True),
            (int8_noconvert, True, False),
            (int16_noconvert, True, False),
            (int32_noconvert, True, False),
            (int64_noconvert, True, False),
            (uint8_noconvert, False, False),
            (uint16_noconvert, False, False),
            (uint32_noconvert, False, False),
            (uint64_noconvert, False, False),
        ):
            with self.subTest(func=func.__name__, arg="int"):
                func(5)
            with self.subTest(func=func.__name__, arg="float"):
                with self.assertRaises(TypeError):
                    func(5.5)
            with self.subTest(func=func.__name__, arg="MyFloat"):
                my_float = MyFloat(5.5)
                if convert:
                    func(my_float)
                else:
                    with self.assertRaises(TypeError):
                        func(my_float)
            with self.subTest(func=func.__name__, arg="MyIndex"):
                my_index = MyIndex(5)
                if convert or signed:
                    func(my_index)
                else:
                    with self.assertRaises(TypeError):
                        func(my_index)
            with self.subTest(func=func.__name__, arg="MyInt"):
                my_int = MyInt(5)
                if convert:
                    func(my_int)
                else:
                    with self.assertRaises(TypeError):
                        func(my_int)
            with self.subTest(func=func.__name__, arg="numpy.float32"):
                np_float = numpy.float32(5.5)
                if convert:
                    func(np_float)
                else:
                    with self.assertRaises(TypeError):
                        func(np_float)

This is the behaviour I would expect. Noconvert __index__ behaviour is open for discussion

            with self.subTest(func=func.__name__, arg="int"):
                func(5)
            with self.subTest(func=func.__name__, arg="float"):
                with self.assertRaises(TypeError):
                    func(5.5)
            with self.subTest(func=func.__name__, arg="MyFloat"):
                my_float = MyFloat(5.5)
                with self.assertRaises(TypeError):
                    func(my_float)
            with self.subTest(func=func.__name__, arg="MyIndex"):
                my_index = MyIndex(5)
                func(my_index)
            with self.subTest(func=func.__name__, arg="MyInt"):
                my_int = MyInt(5.5)
                with self.assertRaises(TypeError):
                    func(my_int)
            with self.subTest(func=func.__name__, arg="numpy.float32"):
                np_float = numpy.float32(5.5)
                with self.assertRaises(TypeError):
                    func(np_float)

Is this a regression? Put the last known working version here if it is.

Not a regression

Start with the integral caster in include/pybind11/cast.h and the unsigned casting helpers in include/pybind11/pytypes.h, then run the provided C++ bindings and Python reproducer across signed and unsigned types. The issue needs an agreed conversion and noconvert policy before implementation; done means the behavior and type hints consistently enforce that policy, including unsigned handling.