pybind / pybind/pybind11

Prevent lossy conversions

Open
#1,305 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Issue description

Pybind11 by default permits lossy conversions from numpy arithmetic types to c++ ones.

I understand the reasoning behind the current situation: that python doesn't really have numeric types other than "float" and "long", and so conversion from python numeric types is always going to be potentially truncating.

However, for numpy scalars (int64, float64, etc.), there is a hierarchy of precision that has direct matches to c++ types. For the work I'm doing accidental loss of precision can be dangerous, so with the boost::python bindings we're looking to migrate away from we've gone to extensive trouble to avoid truncating casts of numpy values from python to c++.

I am aware that for py::array_t I can disable forcecasting (aside, I agree with the discussion in https://github.com/pybind/pybind11/issues/338 that probably no forecasting is a better default).

However, for wrapped functions that take e.g. int32_t or std::vector<int32_t>, then truncation of numpy values can occur.

I think that making the type_caster for numeric types aware of numpy scalars would make it possible to prevent e.g. np.int64 successfully casting to int32_t.

Would such a change be possible? I think this has to be done within the pybind11 cast.h header itself.

With pybind11's first-class support for numpy this feels like it would be a great addition.

Thanks
David

Reproducible example code

For example, this binding code:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
PYBIND11_MODULE(pybind_numpy, m)
{
  m.def (
    "test_takes_int32",
    [] (const int32_t i)
    {
      return i;
    });
  m.def (
    "test_takes_vec_int32",
    [] (const std::vector<int32_t> vec)
    {
      return vec;
    });
}

Can be called like so, but to avoid truncating conversions we would need both of these calls to be TypeErrors:

import pybind_numpy
import numpy as np

# scalar converting to lower-precision scalar
print(pybind_numpy.test_takes_int32(np.int64(1000)))  

# array is iterated by pybind11 and contained scalars being cast to lower-precision scalars
print(pybind_numpy.test_takes_vec_int32(np.array([1000, 2000], dtype=np.int64)))   

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the scalar and vector examples from the issue, then inspect the numeric type_caster in the cast.h header. Determine how NumPy scalar types are currently converted and define tests for the requested behavior. Done means both np.int64-to-int32 cases raise TypeError instead of silently truncating.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, numpy, python
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.