numpy / numpy/numpy

[BUG] numpy implements PEP-3118 incorrectly

Open
#9,456 22 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

PEP 3118 describes Buffers protocol to implement data interchange between different Python/C objects.

I'm implementing such an object, which exposes PyBuffers interface for the purpose of interoperability with other libraries. For example, I expect that if x is my object, then numpy.array(x) would convert it into a numpy array via the Buffers protocol.

What I find however is that numpy.array() constructor calls PyObject_GetBuffer() on my object with flags=0x11C (which is a combination of PyBUF_INDIRECT | PyBUF_FORMAT). According to the specification, the PyBUF_INDIRECT should be used

... when the consumer can handle indirect array referencing implied by these suboffsets.

Unfortunately numpy does not support arrays with suboffsets, and therefore either displays incorrect information or causes a seg.fault when trying to read memory without properly following the pointers as required by the spec.

The expected behavior is that numpy would not supply the PyBUF_INDIRECT flag when calling PyObject_GetBuffer, but instead use the PyBUF_STRIDES flag which seems to correspond to what numpy can actually handle.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the PEP 3118 buffer protocol and the numpy.array() buffer-acquisition path described in the issue. Reproduce the behavior with an object exposing suboffsets, then verify that unsupported indirect buffers are not requested or read incorrectly and add coverage for the resulting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.