[BUG] numpy implements PEP-3118 incorrectly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
PEP 3118 describes Buffers protocol to implement data interchange between different Python/C objects.
I'm implementing such an object, which exposes PyBuffers interface for the purpose of interoperability with other libraries. For example, I expect that if x is my object, then numpy.array(x) would convert it into a numpy array via the Buffers protocol.
What I find however is that numpy.array() constructor calls PyObject_GetBuffer() on my object with flags=0x11C (which is a combination of PyBUF_INDIRECT | PyBUF_FORMAT). According to the specification, the PyBUF_INDIRECT should be used
... when the consumer can handle indirect array referencing implied by these suboffsets.
Unfortunately numpy does not support arrays with suboffsets, and therefore either displays incorrect information or causes a seg.fault when trying to read memory without properly following the pointers as required by the spec.
The expected behavior is that numpy would not supply the PyBUF_INDIRECT flag when calling PyObject_GetBuffer, but instead use the PyBUF_STRIDES flag which seems to correspond to what numpy can actually handle.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the PEP 3118 buffer protocol and the numpy.array() buffer-acquisition path described in the issue. Reproduce the behavior with an object exposing suboffsets, then verify that unsupported indirect buffers are not requested or read incorrectly and add coverage for the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100