numpy.load cannot read from tar archive in python3 ('_FileInFile' object has no attribute 'fileno')
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
I would like to numpy.load an ndarray from a file within a tar archive, without extracting to disk, in Python3.
import os
import numpy
import tarfile
# create a test tar archive
a = numpy.random.rand(10, 4)
numpy.save('a.npy', a)
with open('foo.txt', 'w') as f:
f.write('hey\nho\n')
t = tarfile.open('abc.tar', 'w')
t.add('a.npy')
t.add('foo.txt')
t.close()
del t, a
os.remove('a.npy')
os.remove('foo.txt')
# -----
t = tarfile.open('abc.tar', 'r')
a = numpy.load(t.extractfile('a.npy'))
The above code snippet works fine in Python 2.7.12 with numpy 1.11.1, but fails in Python 3.5.2 with numpy 1.11.1 (Debian Linux):
Traceback (most recent call last):
File "loadtest.py", line 21, in <module>
a = numpy.load(t.extractfile('a.npy'))
File "[...]/lib/python3.5/site-packages/numpy/lib/npyio.py", line 406, in load
pickle_kwargs=pickle_kwargs)
File "[...]/lib/python3.5/site-packages/numpy/lib/format.py", line 648, in read_array
array = numpy.fromfile(fp, dtype=dtype, count=count)
AttributeError: '_FileInFile' object has no attribute 'fileno'
Am I doing something wrong? The numpy docs say that I can pass a file-like object to numpy.load, which must support the seek() and read() methods, which the io.BufferedReader I receive from TarFile.extractfile() does.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproducer in the issue and inspect numpy/lib/npyio.py and numpy/lib/format.py at the failing numpy.load and numpy.fromfile calls. Run the Python 3 tarfile example first, then verify that loading the array from TarFile.extractfile() succeeds without extracting it to disk.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100