ENH: memmap should allow a `strides` kwarg (was `int` limit in `dtype` size)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
I have a binary file that I want to read in with a python routine. In order to do so a dtype object is created, which describes how the data looks like. The dtype object that should be created is a dictionary of the form {'field1': ..., 'field2': ..., ...}. The obj then is a tuple of (data-type, offset) - (from numpy documentation). The error now occurs if offset exceeds the range of a C int during creation of the dtype. I also wrote a similar post regarding this issue on StackOverflow.
Reproducing code example:
import numpy as np
dict_tmp = dict()
offset = 2281832888
dict_tmp['/timedisc/pressure'] = ('(4096, 4096)>f8', offset)
dtype = np.dtype(dict_tmp)
If I reduce the offset below the range of a 32bit integer the error vanishes of course. I already tried to cast the offset value to an int64 or uint32 by hand, but this was also not working. As far as I can see the dtype is part of multiarray in numpy and at this point I am a bit lost.
Error message:
ValueError Traceback (most recent call last)
in ()
2 offset = 2281832888
3 dict_tmp['/timedisc/pressure'] = ('(4096, 4096)>f8', offset)
----> 4 dtype = np.dtype(dict_tmp)/home/jklee/src/anaconda3/lib/python3.5/site-packages/numpy/core/_internal.py in _usefields(adict, align)
77 "formats": formats,
78 "offsets": offsets,
---> 79 "titles": titles}, align)
80
81ValueError: integer won't fit into a C int
Numpy/Python version information:
1.13.1 3.5.4 |Anaconda custom (64-bit)| (default, Aug 14 2017, 13:26:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the np.dtype reproduction and the traceback entry point in numpy/core/_internal.py. Compare the title's memmap/strides request with the reported large dtype offset failure, then inspect related tests and current dtype field handling. Done means the intended large-offset or strides behavior is defined and covered by regression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100