numpy / numpy/numpy

_dtype_from_pep3118 is overly strict on prefixes

Open
#9,049 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug component: numpy._core
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

In numpy 1.12.1, _dtype_from_pep3118 requires that modifiers are specified in the order shape then byte order then count, with at most one of each. The PEP is a bit vague, but it seems like modifiers ought to be interpreted recursively i.e. a modifier can be followed by any valid PEP 3118 specifier.

In practice this can lead to certain types failing to round-trip through a memoryview e.g.:

>>> dt = np.dtype({'formats': [np.dtype((np.dtype((np.int32, (3,))), (2,)))], 'names': ['foo']})
>>> a = np.empty(0, dt)
>>> m = memoryview(a)
>>> np.array(m)
NotImplementedError: memoryview: unsupported format T{(2)(3)i:foo:}

It also causes problems if one puts an endianness specifier at the start of the string (which should always be permitted, since the original struct module allows it). I haven't managed to get numpy to generate such a format string when wrapping an array into a memoryview, but it's causing me some problems while developing new features for pybind11:

>>> np.core._internal._dtype_from_pep3118('<(3)i')
ValueError: Unknown PEP 3118 data type specifier '(3)i'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with NumPy's _dtype_from_pep3118 entry point and reproduce the nested-shape and leading-endianness examples from the issue. Compare its accepted modifier order with the PEP 3118 and struct format rules; done means these formats parse successfully and the memoryview round-trip no longer raises NotImplementedError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.