pybind / pybind/pybind11

Wrong overload picked for numpy 1D arrays

Open
#2,027 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Issue description

When an overloaded function has one version that takes a vector and another that takes a numpy array, the one that takes a vector can be called if a one-dimensional numpy array is passed as argument.

Reproducible example code

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/numpy.h>
#include <vector>
#include <iostream>

namespace py = pybind11;

class Tensor {
public:
    int size;
    std::vector<int> shape;
    float *data;
    Tensor(const std::vector<int>& shape) {
	this->shape = shape;
	size = 1;
	for (size_t i = 0; i < shape.size(); ++i) {
	    size *= shape[i];
	}
	data = new float[size];
    }
    void info() {
	std::cout << "shape: [ ";
	for (size_t i = 0; i < shape.size(); ++i) {
	    std::cout << shape[i] << " ";
	}
	std::cout << "]" << std::endl;
	std::cout << "data: [ ";
	for (int i = 0; i < size; ++i) {
	    std::cout << data[i] << " ";
	}
	std::cout << "]" << std::endl;
    }
};

Tensor* create(const std::vector<int>& shape) {
    return new Tensor(shape);
}

PYBIND11_MODULE(_ext, m) {
    py::class_<Tensor>(m, "Tensor")
	.def(py::init<const std::vector<int>&>())
	.def("info", &Tensor::info);
    m.def("create", (class Tensor* (*)(const std::vector<int>&)) &create,
	  "create(list) --> Tensor", pybind11::arg("shape"));
    m.def("create", [](py::array_t<float, py::array::c_style | py::array::forcecast> array) -> class Tensor* {
        py::buffer_info info = array.request();
        std::vector<int> shape(info.shape.begin(), info.shape.end());
        Tensor* t = new Tensor(shape);
        std::copy((float*)info.ptr, ((float*)info.ptr) + t->size, t->data);
        return t;
    }, "create(array) --> Tensor", py::arg("array"));
}

If a 1D array is passed to create, the vector-based version is picked instead of the array-based one:

import _ext
import numpy as np

t = _ext.create([2, 2])
t.info()
print()

a = np.arange(6).reshape(2, 3).astype(np.float32)
t = _ext.create(a)
t.info()
print()

b = np.array([1, 2]).astype(np.float32)
u = _ext.create(b)
u.info()

Output:

shape: [ 2 2 ]
data: [ 1.22025e+21 4.57678e-41 1.22025e+21 4.57678e-41 ]

shape: [ 2 3 ]
data: [ 0 1 2 3 4 5 ]

shape: [ 1 2 ]
data: [ 1.62673e-37 0 ]

The expected output for the third call was:

shape: [ 2 ]
data: [ 1 2 ]

A workaround is to switch the order of the "create" bindings, so that the array-based one is tried first. However, unless this is intended behavior for some reason, the vector-based version should never be picked if a numpy array is passed. Moreover, the workaround only works if the array is of type np.float32, otherwise the vector-based version is still picked (forcecast seems to have no effect).

This might be related to #1392.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue using the shown create overloads and the 1D NumPy input; inspect pybind11/numpy.h and the overload-dispatch behavior, with #1392 as related context. Done means a 1D NumPy array selects the array overload, including non-float32 arrays, while list inputs still select the vector overload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, numpy, python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.