numpy / numpy/numpy

ENH: memmap should allow a `strides` kwarg (was `int` limit in `dtype` size)

Open
#11,869 27 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

60 - Major release
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

I have a binary file that I want to read in with a python routine. In order to do so a dtype object is created, which describes how the data looks like. The dtype object that should be created is a dictionary of the form {'field1': ..., 'field2': ..., ...}. The obj then is a tuple of (data-type, offset) - (from numpy documentation). The error now occurs if offset exceeds the range of a C int during creation of the dtype. I also wrote a similar post regarding this issue on StackOverflow.

Reproducing code example:
import numpy as np

dict_tmp = dict()
offset = 2281832888
dict_tmp['/timedisc/pressure'] = ('(4096, 4096)>f8', offset)
dtype = np.dtype(dict_tmp)

If I reduce the offset below the range of a 32bit integer the error vanishes of course. I already tried to cast the offset value to an int64 or uint32 by hand, but this was also not working. As far as I can see the dtype is part of multiarray in numpy and at this point I am a bit lost.

Error message:

ValueError Traceback (most recent call last)
in ()
2 offset = 2281832888
3 dict_tmp['/timedisc/pressure'] = ('(4096, 4096)>f8', offset)
----> 4 dtype = np.dtype(dict_tmp)

/home/jklee/src/anaconda3/lib/python3.5/site-packages/numpy/core/_internal.py in _usefields(adict, align)
77 "formats": formats,
78 "offsets": offsets,
---> 79 "titles": titles}, align)
80
81

ValueError: integer won't fit into a C int

Numpy/Python version information:

1.13.1 3.5.4 |Anaconda custom (64-bit)| (default, Aug 14 2017, 13:26:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the np.dtype reproduction and the traceback entry point in numpy/core/_internal.py. Compare the title's memmap/strides request with the reported large dtype offset failure, then inspect related tests and current dtype field handling. Done means the intended large-offset or strides behavior is defined and covered by regression tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.